You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/27 04:20:14 UTC

[GitHub] [spark] AngersZhuuuu opened a new pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

AngersZhuuuu opened a new pull request #29881:
URL: https://github.com/apache/spark/pull/29881


   ### What changes were proposed in this pull request?
   Support `spark.sql.hive.metastore.jars` use HDFS location.
   
   When user need to use path to set hive metastore jars, you should set 
   `spark.sql.hive.metasstore.jars=path` and set real path in `spark.sql.hive.metastore.jars.path`
   since in origin we use `File.pathSeperator` to split path, but `FIle.pathSeparator` is `:` in unix, it will split hdfs location `hdfs://nameservice/xx`. So add new config `spark.sql.hive.metastore.jars.path` to set comma separated paths.
   
   ### Why are the changes needed?
   All spark app can fetch internal version hive jars in HDFS location, not need distribute to all node.
   
   ### Does this PR introduce _any_ user-facing change?
   User can use HDFS location to store hive metastore jars
   
   
   ### How was this patch tested?
   Manuel tested.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705039973


   **[Test build #129521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129521/testReport)** for PR 29881 at commit [`e50050b`](https://github.com/apache/spark/commit/e50050bb77d1a4374ec06c31530e62de755551fe).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700770576






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710745778


   **[Test build #129936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129936/testReport)** for PR 29881 at commit [`6e67c7b`](https://github.com/apache/spark/commit/6e67c7b9fcf9f5c66cb899f5f9000a5598492831).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-707478580


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34331/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708714602


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34370/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705937731






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704346475


   Do we have other configs that support remote files? and how are they implemented?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699589036






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710760994


   **[Test build #129936 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129936/testReport)** for PR 29881 at commit [`6e67c7b`](https://github.com/apache/spark/commit/6e67c7b9fcf9f5c66cb899f5f9000a5598492831).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705965778


   **[Test build #129572 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129572/testReport)** for PR 29881 at commit [`9278404`](https://github.com/apache/spark/commit/927840422002fb76352fcb6beff906a7381b1abb).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r503030946



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +417,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)

Review comment:
       yes (sorry for the syntax).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r503028007



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +417,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
+            }
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil

Review comment:
       > How are these jars be downloaded eventually? we'll need to call `FileSystem` API for that later right?
   
   URLClassLoader will handle this, you can see my previous commit, I use `Utils.fetchFile` to download, but now I remove that.

##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -88,12 +90,23 @@ private[spark] object HiveUtils extends Logging {
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format
+      |   support both local or remote paths.
+      | 4. A classpath in the standard format for both Hive and Hadoop, we should always

Review comment:
       > nit: `we should always` -> `it should always`?
   
   Yea

##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -88,12 +90,23 @@ private[spark] object HiveUtils extends Logging {
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format
+      |   support both local or remote paths.
+      | 4. A classpath in the standard format for both Hive and Hadoop, we should always
+      |   be fully qualified URL to indicate other file systems.
       """.stripMargin)
     .version("1.4.0")
     .stringConf
     .createWithDefault("builtin")
 
+  val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
+    .doc(s"Comma separated path of Hive jars, both support local and remote paths." +

Review comment:
       > nit: `both support` -> `support both`.
   
   DOne




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r505138988



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,23 +440,50 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+      // Convert to files and expand any directories.
+      val jars =
+        HiveUtils.hiveMetastoreJarsPath(sqlConf)
+          .flatMap {
+            case path if Utils.isWindows =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {

Review comment:
       how about current change? Make the logic more simpler.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700527961


   **[Test build #129224 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129224/testReport)** for PR 29881 at commit [`1063a08`](https://github.com/apache/spark/commit/1063a080a4a7e96071841467b112ffa7e9f0365d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704906015






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710751043


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34541/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700691028


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33851/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700666957


   **[Test build #129235 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129235/testReport)** for PR 29881 at commit [`f2869e9`](https://github.com/apache/spark/commit/f2869e9824d035a1279142294fca7cdded5179cf).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699586571


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33756/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710739619


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129933/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r505222509



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format
+      |   support both local or remote paths, it should always be fully qualified URL to indicate
+      |   other file systems.
+      | 4. A classpath in the standard format for both Hive and Hadoop.
       """.stripMargin)
     .version("1.4.0")
     .stringConf
     .createWithDefault("builtin")
 
+  val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
+    .doc(s"Comma separated fully qualified URL of Hive jars, support both local and remote paths," +
+      s"Such as: " +
+      s" 1. file://path/to/jar/xxx.jar" +
+      s" 2. hdfs://nameservice/path/to/jar/xxx.jar" +
+      s" 3. /path/to/jar (path without schema will be treated as HDFS path)" +

Review comment:
       hmm, this depends on the hadoop conf IIUC. e.g. when I run spark locally with default settings, path without scheme is local file system.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700470433


   > Hi, @AngersZhuuuu . Is this PR limited to HDFS? What about `S3` storage?
   
   I will test for  `s3`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710754644


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34541/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709256375


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34437/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r509929134



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path`
+      |   in comma separated format. Support both local or remote paths, it should always
+      |   be fully qualified URL to indicate other file systems.
+      | 4. A classpath in the standard format for both Hive and Hadoop.
       """.stripMargin)
     .version("1.4.0")
     .stringConf
     .createWithDefault("builtin")
 
+  val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
+    .doc(s"Comma separated fully qualified URL of Hive jars, support both local and remote paths," +
+      s"Such as: " +
+      s" 1. file://path/to/jar/xxx.jar" +
+      s" 2. hdfs://nameservice/path/to/jar/xxx.jar" +
+      s" 3. /path/to/jar/ (path without URI scheme follow conf `fs.defaultFS`'s URI schema)" +
+      s" 4. [http/https/ftp]://path/to/jar/xxx.jar" +
+      s"Notice: `http/https/ftp` doesn't support wildcard, but for other URLs support" +

Review comment:
       `but for other ...` -> `but other ...`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-714377832






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709256499






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709285240


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34439/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704876925


   **[Test build #129506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129506/testReport)** for PR 29881 at commit [`0254297`](https://github.com/apache/spark/commit/02542975d8fc6786b223bd403bc179180e5134e7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700527961


   **[Test build #129224 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129224/testReport)** for PR 29881 at commit [`1063a08`](https://github.com/apache/spark/commit/1063a080a4a7e96071841467b112ffa7e9f0365d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504714045



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +419,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {

Review comment:
       > > can we use `DataSource.checkAndGlobPathIfNecessary`?
   > 
   > Yea, I will test this.
   
   @sunchao Follow this suggestion, we can support nested hdfs path wildcard now.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-706868381


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34266/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r505226128



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format
+      |   support both local or remote paths, it should always be fully qualified URL to indicate
+      |   other file systems.
+      | 4. A classpath in the standard format for both Hive and Hadoop.
       """.stripMargin)
     .version("1.4.0")
     .stringConf
     .createWithDefault("builtin")
 
+  val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
+    .doc(s"Comma separated fully qualified URL of Hive jars, support both local and remote paths," +
+      s"Such as: " +
+      s" 1. file://path/to/jar/xxx.jar" +
+      s" 2. hdfs://nameservice/path/to/jar/xxx.jar" +
+      s" 3. /path/to/jar (path without schema will be treated as HDFS path)" +
+      s" 3. [http/https/ftp]://path/to/jar/xxx.jar" +

Review comment:
       This is supported because `IsolatedClientLoader.execJars` supports it?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504444764



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +419,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
+            }
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
+        }
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        HiveUtils.hiveMetastoreJarsPath(sqlConf)
+          .flatMap {
+            case path if Utils.isWindows =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {
+                case "file" | "local" =>
+                  addLocalHiveJars(new File(uri.getPath))
+                case "http" | "https" | "ftp" =>
+                  try {
+                    // validate and fetch URI file
+                    uri.toURL :: Nil
+                  } catch {
+                    case _: Throwable =>
+                      logWarning(s"Hive Jars URI (${uri.toString}) is not a valid URL.")
+                      Nil
+                  }
+                case _ =>
+                  checkRemoteHiveJars(path)
+              }
+          }
+
+      logInfo(
+        s"Initializing HiveMetastoreConnection version $hiveMetastoreVersion " +
+          s"using path: ${jars.mkString(";")}")
+      new IsolatedClientLoader(
+        version = metaVersion,
+        sparkConf = conf,
+        hadoopConf = hadoopConf,
+        execJars = jars.toSeq,

Review comment:
       > So `IsolatedClientLoader` already supports remote JAR paths?
   
   add new config reason is https://github.com/apache/spark/pull/29881#discussion_r496661990




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708902042


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504442666



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +419,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
+            }
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
+        }
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        HiveUtils.hiveMetastoreJarsPath(sqlConf)
+          .flatMap {
+            case path if Utils.isWindows =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {
+                case "file" | "local" =>

Review comment:
       IIUC the Hadoop DFS API also supports local file systems.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504746109



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -102,9 +103,16 @@ private[spark] object HiveUtils extends Logging {
 
   val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
     .doc(s"Comma separated path of Hive jars, support both local and remote paths," +
-      s"we support path wildcards such as `hdfs://path/to/jars/*`, but not support" +
-      s"nested path wildcards such as `hdfs://path/to/jars/*/*`. When ${HIVE_METASTORE_JARS}" +
-      s"is set to `path`, we will use Hive jars configured by this")
+      s"Such as:" +
+      s" 1. /path/to/jar/xxx.jar" +
+      s" 2. file:///path/to/jar/xxx.jar" +
+      s" 3. local:///path/to/jar/xxx.jar" +
+      s" 4. hdfs://path/to/jar/xxx.jar" +

Review comment:
       > how about `s3://`?
   
   As https://github.com/apache/spark/pull/29881#issuecomment-701211241 mentioned. Seems S3's method need call ClassLoader and cause stackoverflow,  I'm not familiar with S3's API,  and maybe we can make a new jira and hope some expert to solve..




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r503035295



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +417,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)

Review comment:
       > Okay. Maybe it is worth pointing out in the description that we only support one single wildcard in the last path component.
   
   Yea, good suggestion, we should make it more clear in desc.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708687737


   **[Test build #129764 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129764/testReport)** for PR 29881 at commit [`b5241c8`](https://github.com/apache/spark/commit/b5241c810377a19d75cafb15c014a5cbe4fbccef).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709177097


   **[Test build #129834 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129834/testReport)** for PR 29881 at commit [`f1a4085`](https://github.com/apache/spark/commit/f1a408575758bd0a03deb0437e619bec9d4846af).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700647001


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129230/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700770576






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r505228738



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,23 +440,43 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+      // Convert to files and expand any directories.
+      val jars =
+        HiveUtils.hiveMetastoreJarsPath(sqlConf)
+          .flatMap {
+            case path if path.contains("\\") && Utils.isWindows =>

Review comment:
       > I don't know windows very well. cc @HyukjinKwon
   
   see https://github.com/apache/spark/blob/77a8efbc05cb4ecc40dd050c363429e71a9f23c1/core/src/main/scala/org/apache/spark/SparkContext.scala#L1902




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699593548






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699583745


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129140/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705108644






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704910911






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504451835



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +419,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {

Review comment:
       > can we use `DataSource.checkAndGlobPathIfNecessary`?
   
   Yea, I will test this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504437785



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -88,12 +90,25 @@ private[spark] object HiveUtils extends Logging {
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format
+      |   support both local or remote paths, it should always be fully qualified URL to indicate
+      |   other file systems.
+      | 4. A classpath in the standard format for both Hive and Hadoop.

Review comment:
       `This property can be one of three options` should be updated to use `four options`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-701765275


   Thank you for checking `S3`. I also believe you can support all remote storages supported by Hadoop. BTW, I'm not sure about the root cause of StackOverflow either.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704955645






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699588173






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705081325






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700646996


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708898260


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-714294262


   **[Test build #130141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130141/testReport)** for PR 29881 at commit [`ea9ef2b`](https://github.com/apache/spark/commit/ea9ef2bbfd8c2314605ae8e4daa12d2b77b31e62).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704903370






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708687992






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r505223985



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,23 +440,43 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+      // Convert to files and expand any directories.
+      val jars =
+        HiveUtils.hiveMetastoreJarsPath(sqlConf)
+          .flatMap {
+            case path if path.contains("\\") && Utils.isWindows =>

Review comment:
       I don't know windows very well. cc @HyukjinKwon 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710754639






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708896336


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34396/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504750732



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,23 +440,50 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+      // Convert to files and expand any directories.
+      val jars =
+        HiveUtils.hiveMetastoreJarsPath(sqlConf)
+          .flatMap {
+            case path if Utils.isWindows =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {

Review comment:
       I don't understand why we need to check the scheme and do things differently. `spark.read.parquet(...)` can support any schema and internally just use the Hadoop DFS API. Can't we do that as well here?
   
   Can you point to other places in Spark that do similar things to support this PR? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704910911


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r496760626



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")

Review comment:
       Path separator is OS dependent.. lets don't change the behaviour




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700621165






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700583541


   **[Test build #129224 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129224/testReport)** for PR 29881 at commit [`1063a08`](https://github.com/apache/spark/commit/1063a080a4a7e96071841467b112ffa7e9f0365d).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708902049


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34396/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-707497354


   **[Test build #129726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129726/testReport)** for PR 29881 at commit [`4016327`](https://github.com/apache/spark/commit/4016327a018cde039433d0b74b4dab2daacad45e).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708897438


   **[Test build #129784 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129784/testReport)** for PR 29881 at commit [`b5241c8`](https://github.com/apache/spark/commit/b5241c810377a19d75cafb15c014a5cbe4fbccef).
    * This patch **fails PySpark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-714294262


   **[Test build #130141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130141/testReport)** for PR 29881 at commit [`ea9ef2b`](https://github.com/apache/spark/commit/ea9ef2bbfd8c2314605ae8e4daa12d2b77b31e62).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r509938592



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path`
+      |   in comma separated format. Support both local or remote paths, it should always
+      |   be fully qualified URL to indicate other file systems.

Review comment:
       > `it should always be fully qualified URL ...` it's not true now.
   
   Remove this now

##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path`
+      |   in comma separated format. Support both local or remote paths, it should always
+      |   be fully qualified URL to indicate other file systems.
+      | 4. A classpath in the standard format for both Hive and Hadoop.
       """.stripMargin)
     .version("1.4.0")
     .stringConf
     .createWithDefault("builtin")
 
+  val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
+    .doc(s"Comma separated fully qualified URL of Hive jars, support both local and remote paths," +
+      s"Such as: " +
+      s" 1. file://path/to/jar/xxx.jar" +
+      s" 2. hdfs://nameservice/path/to/jar/xxx.jar" +
+      s" 3. /path/to/jar/ (path without URI scheme follow conf `fs.defaultFS`'s URI schema)" +
+      s" 4. [http/https/ftp]://path/to/jar/xxx.jar" +
+      s"Notice: `http/https/ftp` doesn't support wildcard, but for other URLs support" +
+      s" nested path wildcard, Such as: " +
+      s" 1. file://path/to/jar/*, file://path/to/jar/*/*" +
+      s" 2. hdfs://nameservice/path/to/jar/*, hdfs://nameservice/path/to/jar/*/*" +

Review comment:
       > ditto, `\n`
   
   Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r503028754



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")
+          .flatMap {
+            case path if path.contains("\\") =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {
+                case null | "file" =>
+                  addLocalHiveJars(new File(uri.getPath))

Review comment:
       > Should this comment on the option "path" instead of "classpath"?
   
   Can't get your point of this comment, can you. make it more clear?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r509928938



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path`
+      |   in comma separated format. Support both local or remote paths, it should always
+      |   be fully qualified URL to indicate other file systems.

Review comment:
       `it should always be fully qualified URL ...` it's not true now.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r496650039



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")
+          .flatMap {
+            case path if path.contains("\\") =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {
+                case null | "file" =>
+                  addLocalHiveJars(new File(uri.getPath))
+                case "local" =>
+                  new File("file:" + uri.getPath).toURL :: Nil

Review comment:
       Here I think you can treat `local` as same as `file` since we're not downloading or fetching it (or distribute)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700583803


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-707459944


   Any more update?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-706470267


   ping @dongjoon-hyun @HyukjinKwon @cloud-fan  Any update?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704903381


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34111/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705953853






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r505252862



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format
+      |   support both local or remote paths, it should always be fully qualified URL to indicate
+      |   other file systems.
+      | 4. A classpath in the standard format for both Hive and Hadoop.
       """.stripMargin)
     .version("1.4.0")
     .stringConf
     .createWithDefault("builtin")
 
+  val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
+    .doc(s"Comma separated fully qualified URL of Hive jars, support both local and remote paths," +
+      s"Such as: " +
+      s" 1. file://path/to/jar/xxx.jar" +
+      s" 2. hdfs://nameservice/path/to/jar/xxx.jar" +
+      s" 3. /path/to/jar (path without schema will be treated as HDFS path)" +
+      s" 3. [http/https/ftp]://path/to/jar/xxx.jar" +

Review comment:
       > This is supported because `IsolatedClientLoader.execJars` supports it?
   
   URLClassLoader support URI path, and `DataSource. checkAndGlobPathIfNecessary` can pass URI too.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699593548






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r505446116



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format
+      |   support both local or remote paths, it should always be fully qualified URL to indicate
+      |   other file systems.
+      | 4. A classpath in the standard format for both Hive and Hadoop.
       """.stripMargin)
     .version("1.4.0")
     .stringConf
     .createWithDefault("builtin")
 
+  val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
+    .doc(s"Comma separated fully qualified URL of Hive jars, support both local and remote paths," +
+      s"Such as: " +
+      s" 1. file://path/to/jar/xxx.jar" +
+      s" 2. hdfs://nameservice/path/to/jar/xxx.jar" +
+      s" 3. /path/to/jar/ (path without schema will follow hadoop conf `fs.defaultFS`'s schema)" +

Review comment:
       > `schema` -> `URI scheme`
   
   Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700582368


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33839/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709309546






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r503029447



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +417,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)

Review comment:
       > Do we handle nested `*`, e.g., `path = `/foo/_/_`?
   
   you mean `path=/xxx/xxx/*/*` ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704900722


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34110/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704751988


   > I think we just need a new configuration that works like SparkContext.addJar/addFile, using `comma separator and resolving URIs with the HDFS's default scheme consistently if we should add this support.
   
   +1. It's a breaking change if `spark.sql.hive.metastore.jars` can't use `:` as separator any more.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705953853


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r496668959



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")
+          .flatMap {
+            case path if path.contains("\\") =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {
+                case null | "file" =>
+                  addLocalHiveJars(new File(uri.getPath))
+                case "local" =>
+                  new File("file:" + uri.getPath).toURL :: Nil

Review comment:
       > Here I think you can treat `local` as same as `file` since we're not downloading or fetching it (or distribute)
   
   Yae




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r496661990



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")

Review comment:
       > Why do we change this from `File.pathSeparator` to `;`?
   
   Since in unix system, `File.pathSeparator` is `:` will split hdfs path `hdfs://xxx/xxx`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699583150


   ping @wangyum 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710748246






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699584677


   **[Test build #129141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129141/testReport)** for PR 29881 at commit [`bf3c4b6`](https://github.com/apache/spark/commit/bf3c4b62f63f2f24945df38e487455004bd2d892).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708857615


   **[Test build #129784 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129784/testReport)** for PR 29881 at commit [`b5241c8`](https://github.com/apache/spark/commit/b5241c810377a19d75cafb15c014a5cbe4fbccef).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709116134


   **[Test build #129832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129832/testReport)** for PR 29881 at commit [`b79ab0d`](https://github.com/apache/spark/commit/b79ab0d93a6e86fbf0706588d21c0540b3a4c0ab).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709325995






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710739403


   **[Test build #129933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129933/testReport)** for PR 29881 at commit [`d631a75`](https://github.com/apache/spark/commit/d631a758e491fb7f5e7ffcb66f74cdef19028be8).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r503035033



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +417,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
+            }
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil

Review comment:
       > Saw the `fetchFile` call in the previous commit. However I don't see how `URLClassLoader` can handle HDFS paths. I don't see any `FileSystem` related stuff in the class.
   
   I have same confuse as you before, until I see this error https://github.com/apache/spark/pull/29881#issuecomment-701211241




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705081325






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704896417


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34112/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #29881:
URL: https://github.com/apache/spark/pull/29881


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704918165






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504443964



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -88,12 +90,25 @@ private[spark] object HiveUtils extends Logging {
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format
+      |   support both local or remote paths, it should always be fully qualified URL to indicate
+      |   other file systems.
+      | 4. A classpath in the standard format for both Hive and Hadoop.

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r503033148



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +417,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
+            }
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil

Review comment:
       Saw the `fetchFile` call in the previous commit. However I don't see how `URLClassLoader` can handle HDFS paths. I don't see any `FileSystem` related stuff in the class. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-707462326


   **[Test build #129726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129726/testReport)** for PR 29881 at commit [`4016327`](https://github.com/apache/spark/commit/4016327a018cde039433d0b74b4dab2daacad45e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704905998


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34112/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r506311427



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format

Review comment:
       Add a period at the end?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700613588


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33846/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700666957


   **[Test build #129235 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129235/testReport)** for PR 29881 at commit [`f2869e9`](https://github.com/apache/spark/commit/f2869e9824d035a1279142294fca7cdded5179cf).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700700361


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704950367






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699588173






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705937731


   **[Test build #129572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129572/testReport)** for PR 29881 at commit [`9278404`](https://github.com/apache/spark/commit/927840422002fb76352fcb6beff906a7381b1abb).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705953853






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704889349


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34111/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r503030308



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")
+          .flatMap {
+            case path if path.contains("\\") =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {
+                case null | "file" =>
+                  addLocalHiveJars(new File(uri.getPath))

Review comment:
       > Should this comment on the option "path" instead of "classpath"?
   
   Can't got your point, can you make it more clear?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r496668116



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")
+          .flatMap {
+            case path if path.contains("\\") =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {
+                case null | "file" =>
+                  addLocalHiveJars(new File(uri.getPath))

Review comment:
       > Assuming `null` as a local path is kind of ambiguous (local paths vs `fs.default.name`). Maybe you'll have to document that it should always be fully qualified URL to indicate other file systems.
   
   Add comment in doc of conf `spark.sql.hive.metastore.jars`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r505254274



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format
+      |   support both local or remote paths, it should always be fully qualified URL to indicate
+      |   other file systems.
+      | 4. A classpath in the standard format for both Hive and Hadoop.
       """.stripMargin)
     .version("1.4.0")
     .stringConf
     .createWithDefault("builtin")
 
+  val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
+    .doc(s"Comma separated fully qualified URL of Hive jars, support both local and remote paths," +
+      s"Such as: " +
+      s" 1. file://path/to/jar/xxx.jar" +
+      s" 2. hdfs://nameservice/path/to/jar/xxx.jar" +
+      s" 3. /path/to/jar (path without schema will be treated as HDFS path)" +
+      s" 3. [http/https/ftp]://path/to/jar/xxx.jar" +

Review comment:
       remove hive-exec jar in hdfs path and bring with URI path can build  HiveClient too.
   ```
   --conf spark.sql.hive.metastore.jars=path --conf spark.sql.hive.metastore.jars.path=hdfs://nameservice/spark/hive-2.3.7/*,https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.7/hive-exec-2.3.7.jar
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-714403701


   **[Test build #130141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130141/testReport)** for PR 29881 at commit [`ea9ef2b`](https://github.com/apache/spark/commit/ea9ef2bbfd8c2314605ae8e4daa12d2b77b31e62).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-706911022






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710761219






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710754632


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34541/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704949414






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r503041748



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +417,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
+            }
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil

Review comment:
       > I see. So Hadoop side has a `URLStreamHandlerFactory` impl which provides handlers for the Hadoop-specific URL schemes. Not sure about the S3 error though :-/
   
   Yea,  dig into the code, seems s3 find config will call URLClassLoader and not equal make it reload element again and again.  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r496646841



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")
+          .flatMap {
+            case path if path.contains("\\") =>

Review comment:
       I could not follow this. Do you want to check if this is a Windows path? If that's the case, you can use `Utils.isWindows`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704657971


   > Do we have other configs that support remote files? and how are they implemented?
   
   Add `add jar / add file SQL command`,conf such as  `spark.jars`, and it use  `SparkContext.addJar/addFile` support remote file. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705953853






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r505224964



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format
+      |   support both local or remote paths, it should always be fully qualified URL to indicate
+      |   other file systems.
+      | 4. A classpath in the standard format for both Hive and Hadoop.
       """.stripMargin)
     .version("1.4.0")
     .stringConf
     .createWithDefault("builtin")
 
+  val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
+    .doc(s"Comma separated fully qualified URL of Hive jars, support both local and remote paths," +
+      s"Such as: " +
+      s" 1. file://path/to/jar/xxx.jar" +
+      s" 2. hdfs://nameservice/path/to/jar/xxx.jar" +
+      s" 3. /path/to/jar (path without schema will be treated as HDFS path)" +

Review comment:
       > hmm, this depends on the hadoop conf IIUC. e.g. when I run spark locally with default settings, path without scheme is local file system.
   
   emmm,  I need to change the doc..




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-714377844


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34748/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705966172






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700594321






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r496775069



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")

Review comment:
       > Path separator is OS dependent.. lets don't change the behaviour
   
   But a lot place we use comma separator to split file path, right? Also I am confused that why we must need to use this Path separator here since what we config is jar file list or `*`.
   
   or you mean use will pass config like `hadoop classpath`, but seem no `hive classpath` 
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700702718


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33851/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704903370


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708687992


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700593025


   **[Test build #129230 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129230/testReport)** for PR 29881 at commit [`6214ebf`](https://github.com/apache/spark/commit/6214ebf03855b76d84990df3f965941287b1d3d3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-706849086


   **[Test build #129662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129662/testReport)** for PR 29881 at commit [`9ee1a86`](https://github.com/apache/spark/commit/9ee1a861e650f716870befcefd9820dc8425b9ae).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700700374


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33852/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-706875583






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704918130


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34114/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704877999


   > +1. It's a breaking change if `spark.sql.hive.metastore.jars` can't use `:` as separator any more.
   
   @HyukjinKwon @cloud-fan How about latest change


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-701780166


   > Thank you for checking `S3`. I also believe you can support all remote storages supported by Hadoop. BTW, I'm not sure about the root cause of StackOverflow either.
   
   Use `Utils.fetchFile` to download remote file and return tmp path can working well, should we do like this?
   
   
   Also I will try find root cause about this StackOverflow


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709325995






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704950367






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704949414






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705081301


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34126/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705937731


   **[Test build #129572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129572/testReport)** for PR 29881 at commit [`9278404`](https://github.com/apache/spark/commit/927840422002fb76352fcb6beff906a7381b1abb).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700594290


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33840/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704900759


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34110/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700702757






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700584526


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33840/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504444764



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +419,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
+            }
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
+        }
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        HiveUtils.hiveMetastoreJarsPath(sqlConf)
+          .flatMap {
+            case path if Utils.isWindows =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {
+                case "file" | "local" =>
+                  addLocalHiveJars(new File(uri.getPath))
+                case "http" | "https" | "ftp" =>
+                  try {
+                    // validate and fetch URI file
+                    uri.toURL :: Nil
+                  } catch {
+                    case _: Throwable =>
+                      logWarning(s"Hive Jars URI (${uri.toString}) is not a valid URL.")
+                      Nil
+                  }
+                case _ =>
+                  checkRemoteHiveJars(path)
+              }
+          }
+
+      logInfo(
+        s"Initializing HiveMetastoreConnection version $hiveMetastoreVersion " +
+          s"using path: ${jars.mkString(";")}")
+      new IsolatedClientLoader(
+        version = metaVersion,
+        sparkConf = conf,
+        hadoopConf = hadoopConf,
+        execJars = jars.toSeq,

Review comment:
       > So `IsolatedClientLoader` already supports remote JAR paths?
   
   https://github.com/apache/spark/pull/29881#discussion_r496661990




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708898265


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129784/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708884130


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129789/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700593025


   **[Test build #129230 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129230/testReport)** for PR 29881 at commit [`6214ebf`](https://github.com/apache/spark/commit/6214ebf03855b76d84990df3f965941287b1d3d3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708722064






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705966172






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699593430


   **[Test build #129141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129141/testReport)** for PR 29881 at commit [`bf3c4b6`](https://github.com/apache/spark/commit/bf3c4b62f63f2f24945df38e487455004bd2d892).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-706875583






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504442190



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +419,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {

Review comment:
       can we use `DataSource.checkAndGlobPathIfNecessary`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704876925


   **[Test build #129506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129506/testReport)** for PR 29881 at commit [`0254297`](https://github.com/apache/spark/commit/02542975d8fc6786b223bd403bc179180e5134e7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704906026


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34112/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708871976


   **[Test build #129789 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129789/testReport)** for PR 29881 at commit [`6fbe082`](https://github.com/apache/spark/commit/6fbe082416a778d1b71bf3c5560f8b9725938c6a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r506310103



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,23 +440,43 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+      // Convert to files and expand any directories.
+      val jars =
+        HiveUtils.hiveMetastoreJarsPath(sqlConf)
+          .flatMap {
+            case path if path.contains("\\") && Utils.isWindows =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              DataSource.checkAndGlobPathIfNecessary(
+                pathStrings = Seq(path),
+                hadoopConf = hadoopConf,
+                checkEmptyGlobPath = true,
+                checkFilesExist = false,
+                enableGlobbing = true
+              ).map(_.toUri.toURL)
+          }
+
+      logInfo(
+        s"Initializing HiveMetastoreConnection version $hiveMetastoreVersion " +
+          s"using path: ${jars.mkString(";")}")
+      new IsolatedClientLoader(
+        version = metaVersion,
+        sparkConf = conf,
+        hadoopConf = hadoopConf,
+        execJars = jars.toSeq,
+        config = configurations,
+        isolationOn = true,
+        barrierPrefixes = hiveMetastoreBarrierPrefixes,
+        sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
       // Convert to files and expand any directories.
       val jars =
         hiveMetastoreJars
           .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
-              Nil
-            } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
-            }
-          case path =>
-            new File(path) :: Nil
-        }
+          .flatMap { path =>
+            addLocalHiveJars(new File(path))
+          }
           .map(_.toURI.toURL)

Review comment:
       This line can be removed, as `addLocalHiveJars` already returns URL




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708884000


   **[Test build #129789 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129789/testReport)** for PR 29881 at commit [`6fbe082`](https://github.com/apache/spark/commit/6fbe082416a778d1b71bf3c5560f8b9725938c6a).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704880277


   **[Test build #129508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129508/testReport)** for PR 29881 at commit [`c6475ee`](https://github.com/apache/spark/commit/c6475ee20e85f981322e75a410cbf803a8c6d5c6).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r496649487



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")
+          .flatMap {
+            case path if path.contains("\\") =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {
+                case null | "file" =>
+                  addLocalHiveJars(new File(uri.getPath))

Review comment:
       Assuming `null` as a local path is kind of ambiguous (local paths vs `fs.default.name`). Maybe you'll have to document that it should always be fully qualified URL to indicate other file systems.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708886296






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700621150


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33846/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700591703


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33839/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710739618






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r506787897



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format

Review comment:
       > Add a period at the end?
   
   Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709300166


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34439/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700688943


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33852/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705107723


   **[Test build #129521 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129521/testReport)** for PR 29881 at commit [`e50050b`](https://github.com/apache/spark/commit/e50050bb77d1a4374ec06c31530e62de755551fe).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709177097


   **[Test build #129834 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129834/testReport)** for PR 29881 at commit [`f1a4085`](https://github.com/apache/spark/commit/f1a408575758bd0a03deb0437e619bec9d4846af).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708902031


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34396/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710748237


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34538/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709300205






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699583671


   **[Test build #129140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129140/testReport)** for PR 29881 at commit [`834c2a7`](https://github.com/apache/spark/commit/834c2a77a9a36e39f67eafd07def9b7e9f47639e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu edited a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu edited a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704657971


   > Do we have other configs that support remote files? and how are they implemented?
   
   Add `add jar / add file SQL command`,conf such as  `spark.jars`, and it use  `SparkContext.addJar/addFile` support remote file. 
   
   And I implement this pr use similar way like `SparkContext.addJar`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709231762


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34437/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708898260






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504444397



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -88,12 +90,25 @@ private[spark] object HiveUtils extends Logging {
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format

Review comment:
       > `${HIVE_METASTORE_JARS_PATH.key}`
   
   We can't refer to each other between `HIVE_METASTORE_JARS_PATH` and `HIVE_METASTORE_JARS`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700702757






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r496645332



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")

Review comment:
       Why do we change this from `File.pathSeparator` to `;`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700646248


   **[Test build #129230 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129230/testReport)** for PR 29881 at commit [`6214ebf`](https://github.com/apache/spark/commit/6214ebf03855b76d84990df3f965941287b1d3d3).
    * This patch **fails PySpark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700583812


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129224/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700646996






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705039973


   **[Test build #129521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129521/testReport)** for PR 29881 at commit [`e50050b`](https://github.com/apache/spark/commit/e50050bb77d1a4374ec06c31530e62de755551fe).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700769452


   **[Test build #129235 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129235/testReport)** for PR 29881 at commit [`f2869e9`](https://github.com/apache/spark/commit/f2869e9824d035a1279142294fca7cdded5179cf).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705108644






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708879616


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34391/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-701211241


   > Hi, @AngersZhuuuu . Is this PR limited to HDFS? What about `S3` storage?
   
   Hi @dongjoon-hyun , when I use `s3` path, it can pass check and get listed file paths, bug I am confused about this  `StackOverflowError`, it seems not normal fetch `s3` data file error, but a error when ClassLoader fetch `s3` URL.
   I am not so familiar with `s3` ,can you give some advise?
   
   if we use `Utils.fetchFIle` and return a local path seems will be ok.
   
   
   ```
   Exception in thread "main" java.lang.StackOverflowError
   	at java.security.AccessController.doPrivileged(Native Method)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:554)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
   	at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484)
   	at sun.misc.URLClassPath.access$100(URLClassPath.java:65)
   	at sun.misc.URLClassPath$1.next(URLClassPath.java:266)
   	at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277)
   	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
   	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
   	at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
   	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
   	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
   	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
   	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getConfigurationFile(LogFactory.java:1409)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getFactory(LogFactory.java:455)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getLog(LogFactory.java:657)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpResponseParser.<init>(DefaultHttpResponseParser.java:59)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpResponseParserFactory.create(DefaultHttpResponseParserFactory.java:76)
   	at com.amazonaws.thirdparty.apache.http.impl.DefaultBHttpClientConnection.<init>(DefaultBHttpClientConnection.java:99)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultManagedHttpClientConnection.<init>(DefaultManagedHttpClientConnection.java:74)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.LoggingManagedHttpClientConnection.<init>(LoggingManagedHttpClientConnection.java:66)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.ManagedHttpClientConnectionFactory.create(ManagedHttpClientConnectionFactory.java:127)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.ManagedHttpClientConnectionFactory.create(ManagedHttpClientConnectionFactory.java:57)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.create(PoolingHttpClientConnectionManager.java:619)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.create(PoolingHttpClientConnectionManager.java:590)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:363)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:246)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:303)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
   	at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
   	at com.amazonaws.http.conn.$Proxy27.get(Unknown Source)
   	at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
   	at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   	at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1238)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
   	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
   	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4368)
   	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4315)
   	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1271)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$4(S3AFileSystem.java:1290)
   	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
   	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:285)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1287)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2224)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2204)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2143)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:716)
   	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
   	at org.apache.hadoop.fs.FsUrlConnection.connect(FsUrlConnection.java:60)
   	at org.apache.hadoop.fs.FsUrlConnection.getInputStream(FsUrlConnection.java:69)
   	at sun.net.www.protocol.jar.URLJarFile.retrieve(URLJarFile.java:214)
   	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:71)
   	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:84)
   	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
   	at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
   	at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:934)
   	at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791)
   	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876)
   	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868)
   	at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:819)
   	at sun.misc.URLClassPath$3.run(URLClassPath.java:565)
   	at sun.misc.URLClassPath$3.run(URLClassPath.java:555)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:554)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
   	at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484)
   	at sun.misc.URLClassPath.access$100(URLClassPath.java:65)
   	at sun.misc.URLClassPath$1.next(URLClassPath.java:266)
   	at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277)
   	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
   	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
   	at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
   	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
   	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
   	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
   	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getConfigurationFile(LogFactory.java:1409)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getFactory(LogFactory.java:455)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getLog(LogFactory.java:657)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpResponseParser.<init>(DefaultHttpResponseParser.java:59)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpResponseParserFactory.create(DefaultHttpResponseParserFactory.java:76)
   	at com.amazonaws.thirdparty.apache.http.impl.DefaultBHttpClientConnection.<init>(DefaultBHttpClientConnection.java:99)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultManagedHttpClientConnection.<init>(DefaultManagedHttpClientConnection.java:74)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.LoggingManagedHttpClientConnection.<init>(LoggingManagedHttpClientConnection.java:66)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.ManagedHttpClientConnectionFactory.create(ManagedHttpClientConnectionFactory.java:127)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.ManagedHttpClientConnectionFactory.create(ManagedHttpClientConnectionFactory.java:57)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.create(PoolingHttpClientConnectionManager.java:619)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.create(PoolingHttpClientConnectionManager.java:590)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:363)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:246)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:303)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
   	at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
   	at com.amazonaws.http.conn.$Proxy27.get(Unknown Source)
   	at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
   	at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   	at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1238)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
   	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
   	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4368)
   	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4315)
   	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1271)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$4(S3AFileSystem.java:1290)
   	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
   	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:285)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1287)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2224)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2204)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2143)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:716)
   	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
   	at org.apache.hadoop.fs.FsUrlConnection.connect(FsUrlConnection.java:60)
   	at org.apache.hadoop.fs.FsUrlConnection.getInputStream(FsUrlConnection.java:69)
   	at sun.net.www.protocol.jar.URLJarFile.retrieve(URLJarFile.java:214)
   	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:71)
   	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:84)
   	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
   	at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
   	at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:934)
   	at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791)
   	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876)
   	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868)
   	at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:819)
   	at sun.misc.URLClassPath$3.run(URLClassPath.java:565)
   	at sun.misc.URLClassPath$3.run(URLClassPath.java:555)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:554)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
   	at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484)
   	at sun.misc.URLClassPath.access$100(URLClassPath.java:65)
   	at sun.misc.URLClassPath$1.next(URLClassPath.java:266)
   	at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277)
   	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
   	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
   	at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
   	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
   	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
   	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
   	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getConfigurationFile(LogFactory.java:1409)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getFactory(LogFactory.java:455)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getLog(LogFactory.java:657)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpResponseParser.<init>(DefaultHttpResponseParser.java:59)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpResponseParserFactory.create(DefaultHttpResponseParserFactory.java:76)
   	at com.amazonaws.thirdparty.apache.http.impl.DefaultBHttpClientConnection.<init>(DefaultBHttpClientConnection.java:99)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultManagedHttpClientConnection.<init>(DefaultManagedHttpClientConnection.java:74)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.LoggingManagedHttpClientConnection.<init>(LoggingManagedHttpClientConnection.java:66)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.ManagedHttpClientConnectionFactory.create(ManagedHttpClientConnectionFactory.java:127)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.ManagedHttpClientConnectionFactory.create(ManagedHttpClientConnectionFactory.java:57)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.create(PoolingHttpClientConnectionManager.java:619)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.create(PoolingHttpClientConnectionManager.java:590)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:363)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:246)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:303)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
   	at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
   	at com.amazonaws.http.conn.$Proxy27.get(Unknown Source)
   	at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
   	at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   	at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1238)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)
   
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704908677


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34114/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699588171


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33756/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r503034791



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +417,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)

Review comment:
       Okay. Maybe it is worth pointing out in the description that we only support one single wildcard in the last path component.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-707478589


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-707478598


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34331/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709300205






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708884125






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-706849086


   **[Test build #129662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129662/testReport)** for PR 29881 at commit [`9ee1a86`](https://github.com/apache/spark/commit/9ee1a861e650f716870befcefd9820dc8425b9ae).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700594321






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699583741


   **[Test build #129140 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129140/testReport)** for PR 29881 at commit [`834c2a7`](https://github.com/apache/spark/commit/834c2a77a9a36e39f67eafd07def9b7e9f47639e).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r496760626



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")

Review comment:
       File separator is OS dependent.. lets don't change the behaviour




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700583803






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r506787887



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,23 +440,43 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+      // Convert to files and expand any directories.
+      val jars =
+        HiveUtils.hiveMetastoreJarsPath(sqlConf)
+          .flatMap {
+            case path if path.contains("\\") && Utils.isWindows =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              DataSource.checkAndGlobPathIfNecessary(
+                pathStrings = Seq(path),
+                hadoopConf = hadoopConf,
+                checkEmptyGlobPath = true,
+                checkFilesExist = false,
+                enableGlobbing = true
+              ).map(_.toUri.toURL)
+          }
+
+      logInfo(
+        s"Initializing HiveMetastoreConnection version $hiveMetastoreVersion " +
+          s"using path: ${jars.mkString(";")}")
+      new IsolatedClientLoader(
+        version = metaVersion,
+        sparkConf = conf,
+        hadoopConf = hadoopConf,
+        execJars = jars.toSeq,
+        config = configurations,
+        isolationOn = true,
+        barrierPrefixes = hiveMetastoreBarrierPrefixes,
+        sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
       // Convert to files and expand any directories.
       val jars =
         hiveMetastoreJars
           .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
-              Nil
-            } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
-            }
-          case path =>
-            new File(path) :: Nil
-        }
+          .flatMap { path =>
+            addLocalHiveJars(new File(path))
+          }
           .map(_.toURI.toURL)

Review comment:
       > This line can be removed, as `addLocalHiveJars` already returns URL
   
   Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708722052


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34370/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709308558


   **[Test build #129832 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129832/testReport)** for PR 29881 at commit [`b79ab0d`](https://github.com/apache/spark/commit/b79ab0d93a6e86fbf0706588d21c0540b3a4c0ab).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704900752


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-707497982






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699587803


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33757/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-700700333


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33852/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r503031298



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +417,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)

Review comment:
       > yes (sorry for the syntax).
   
   Not yet.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504714045



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +419,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {

Review comment:
       > > can we use `DataSource.checkAndGlobPathIfNecessary`?
   > 
   > Yea, I will test this.
   
   @sunchao Follow this suggestion, we can support nested hdfs path wildcard now such as hdfs://xx/xx/*/*




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704886094


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34110/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504717621



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")
+          .flatMap {
+            case path if path.contains("\\") =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {
+                case null | "file" =>
+                  addLocalHiveJars(new File(uri.getPath))

Review comment:
       @HyukjinKwon 
   Add back null since for local path, user won't  add `file://` or `local://`  under subconscious mind




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu edited a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu edited a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-701211241


   > Hi, @AngersZhuuuu . Is this PR limited to HDFS? What about `S3` storage?
   
   Hi @dongjoon-hyun , when I use `s3` path, it can pass check and get listed file paths, but I am confused about this  `StackOverflowError`, it seems not normal fetch `s3` data file error, but a error when ClassLoader fetch `s3` URL.
   I am not so familiar with `s3` ,can you give some advise?
   
   if we use `Utils.fetchFIle` and return a local path seems will be ok.
   
   
   ```
   Exception in thread "main" java.lang.StackOverflowError
   	at java.security.AccessController.doPrivileged(Native Method)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:554)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
   	at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484)
   	at sun.misc.URLClassPath.access$100(URLClassPath.java:65)
   	at sun.misc.URLClassPath$1.next(URLClassPath.java:266)
   	at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277)
   	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
   	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
   	at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
   	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
   	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
   	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
   	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getConfigurationFile(LogFactory.java:1409)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getFactory(LogFactory.java:455)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getLog(LogFactory.java:657)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpResponseParser.<init>(DefaultHttpResponseParser.java:59)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpResponseParserFactory.create(DefaultHttpResponseParserFactory.java:76)
   	at com.amazonaws.thirdparty.apache.http.impl.DefaultBHttpClientConnection.<init>(DefaultBHttpClientConnection.java:99)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultManagedHttpClientConnection.<init>(DefaultManagedHttpClientConnection.java:74)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.LoggingManagedHttpClientConnection.<init>(LoggingManagedHttpClientConnection.java:66)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.ManagedHttpClientConnectionFactory.create(ManagedHttpClientConnectionFactory.java:127)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.ManagedHttpClientConnectionFactory.create(ManagedHttpClientConnectionFactory.java:57)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.create(PoolingHttpClientConnectionManager.java:619)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.create(PoolingHttpClientConnectionManager.java:590)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:363)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:246)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:303)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
   	at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
   	at com.amazonaws.http.conn.$Proxy27.get(Unknown Source)
   	at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
   	at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   	at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1238)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
   	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
   	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4368)
   	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4315)
   	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1271)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$4(S3AFileSystem.java:1290)
   	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
   	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:285)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1287)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2224)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2204)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2143)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:716)
   	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
   	at org.apache.hadoop.fs.FsUrlConnection.connect(FsUrlConnection.java:60)
   	at org.apache.hadoop.fs.FsUrlConnection.getInputStream(FsUrlConnection.java:69)
   	at sun.net.www.protocol.jar.URLJarFile.retrieve(URLJarFile.java:214)
   	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:71)
   	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:84)
   	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
   	at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
   	at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:934)
   	at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791)
   	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876)
   	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868)
   	at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:819)
   	at sun.misc.URLClassPath$3.run(URLClassPath.java:565)
   	at sun.misc.URLClassPath$3.run(URLClassPath.java:555)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:554)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
   	at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484)
   	at sun.misc.URLClassPath.access$100(URLClassPath.java:65)
   	at sun.misc.URLClassPath$1.next(URLClassPath.java:266)
   	at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277)
   	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
   	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
   	at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
   	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
   	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
   	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
   	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getConfigurationFile(LogFactory.java:1409)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getFactory(LogFactory.java:455)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getLog(LogFactory.java:657)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpResponseParser.<init>(DefaultHttpResponseParser.java:59)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpResponseParserFactory.create(DefaultHttpResponseParserFactory.java:76)
   	at com.amazonaws.thirdparty.apache.http.impl.DefaultBHttpClientConnection.<init>(DefaultBHttpClientConnection.java:99)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultManagedHttpClientConnection.<init>(DefaultManagedHttpClientConnection.java:74)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.LoggingManagedHttpClientConnection.<init>(LoggingManagedHttpClientConnection.java:66)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.ManagedHttpClientConnectionFactory.create(ManagedHttpClientConnectionFactory.java:127)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.ManagedHttpClientConnectionFactory.create(ManagedHttpClientConnectionFactory.java:57)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.create(PoolingHttpClientConnectionManager.java:619)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.create(PoolingHttpClientConnectionManager.java:590)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:363)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:246)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:303)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
   	at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
   	at com.amazonaws.http.conn.$Proxy27.get(Unknown Source)
   	at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
   	at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   	at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1238)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
   	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
   	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4368)
   	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4315)
   	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1271)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$4(S3AFileSystem.java:1290)
   	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
   	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:285)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1287)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2224)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2204)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2143)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:716)
   	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
   	at org.apache.hadoop.fs.FsUrlConnection.connect(FsUrlConnection.java:60)
   	at org.apache.hadoop.fs.FsUrlConnection.getInputStream(FsUrlConnection.java:69)
   	at sun.net.www.protocol.jar.URLJarFile.retrieve(URLJarFile.java:214)
   	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:71)
   	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:84)
   	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
   	at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
   	at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:934)
   	at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:791)
   	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:876)
   	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:869)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:868)
   	at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:819)
   	at sun.misc.URLClassPath$3.run(URLClassPath.java:565)
   	at sun.misc.URLClassPath$3.run(URLClassPath.java:555)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:554)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
   	at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:484)
   	at sun.misc.URLClassPath.access$100(URLClassPath.java:65)
   	at sun.misc.URLClassPath$1.next(URLClassPath.java:266)
   	at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:277)
   	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
   	at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
   	at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
   	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
   	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
   	at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
   	at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getConfigurationFile(LogFactory.java:1409)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getFactory(LogFactory.java:455)
   	at com.amazonaws.thirdparty.apache.logging.LogFactory.getLog(LogFactory.java:657)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpResponseParser.<init>(DefaultHttpResponseParser.java:59)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpResponseParserFactory.create(DefaultHttpResponseParserFactory.java:76)
   	at com.amazonaws.thirdparty.apache.http.impl.DefaultBHttpClientConnection.<init>(DefaultBHttpClientConnection.java:99)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultManagedHttpClientConnection.<init>(DefaultManagedHttpClientConnection.java:74)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.LoggingManagedHttpClientConnection.<init>(LoggingManagedHttpClientConnection.java:66)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.ManagedHttpClientConnectionFactory.create(ManagedHttpClientConnectionFactory.java:127)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.ManagedHttpClientConnectionFactory.create(ManagedHttpClientConnectionFactory.java:57)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.create(PoolingHttpClientConnectionManager.java:619)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.create(PoolingHttpClientConnectionManager.java:590)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:363)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:246)
   	at com.amazonaws.thirdparty.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:303)
   	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
   	at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
   	at com.amazonaws.http.conn.$Proxy27.get(Unknown Source)
   	at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
   	at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   	at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1238)
   	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)
   
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708687999


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129764/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710754639


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710761219






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r497952649



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")

Review comment:
       ping @dongjoon-hyun @wangyum @cloud-fan WDYT about this problem




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r505120663



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,23 +440,50 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+      // Convert to files and expand any directories.
+      val jars =
+        HiveUtils.hiveMetastoreJarsPath(sqlConf)
+          .flatMap {
+            case path if Utils.isWindows =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {

Review comment:
       > I don't understand why we need to check the scheme and do things differently. `spark.read.parquet(...)` can support any schema and internally just use the Hadoop DFS API. Can't we do that as well here?
   > 
   > Can you point to other places in Spark that do similar things to support this PR?
   
   In datasource file index, path is fully qualified URL to indicate other file systems.
   So it support  `file://  | local:// `, but without schema, it will be handled as hdfs path.
   One way to avoid check schema,  as https://github.com/apache/spark/pull/29881#discussion_r496649487 mentioned, we host user use fully qualified URL.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704866420


   **[Test build #129504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129504/testReport)** for PR 29881 at commit [`c200d9b`](https://github.com/apache/spark/commit/c200d9b8503360cd0a268d1cb67e8c471df9705e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699583743






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699589033


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33757/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r496662585



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")
+          .flatMap {
+            case path if path.contains("\\") =>

Review comment:
       > `\\` can be in file names in other OSs. I don't think it makes sense to assume that this is a local path.
   
   All right, I will raise a pr to change this in `SparkContext`'s `addJar/addFile`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r503039921



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +417,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
+            }
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil

Review comment:
       I see. So Hadoop side has a `URLStreamHandlerFactory` impl which provides handlers for the Hadoop-specific URL schemes. Not sure about the S3 error though :-/




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r509929511



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path`
+      |   in comma separated format. Support both local or remote paths, it should always
+      |   be fully qualified URL to indicate other file systems.
+      | 4. A classpath in the standard format for both Hive and Hadoop.
       """.stripMargin)
     .version("1.4.0")
     .stringConf
     .createWithDefault("builtin")
 
+  val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
+    .doc(s"Comma separated fully qualified URL of Hive jars, support both local and remote paths," +
+      s"Such as: " +
+      s" 1. file://path/to/jar/xxx.jar" +

Review comment:
       shall we add `\n` at the end for each item?

##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path`
+      |   in comma separated format. Support both local or remote paths, it should always
+      |   be fully qualified URL to indicate other file systems.
+      | 4. A classpath in the standard format for both Hive and Hadoop.
       """.stripMargin)
     .version("1.4.0")
     .stringConf
     .createWithDefault("builtin")
 
+  val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
+    .doc(s"Comma separated fully qualified URL of Hive jars, support both local and remote paths," +
+      s"Such as: " +
+      s" 1. file://path/to/jar/xxx.jar" +
+      s" 2. hdfs://nameservice/path/to/jar/xxx.jar" +
+      s" 3. /path/to/jar/ (path without URI scheme follow conf `fs.defaultFS`'s URI schema)" +
+      s" 4. [http/https/ftp]://path/to/jar/xxx.jar" +
+      s"Notice: `http/https/ftp` doesn't support wildcard, but for other URLs support" +
+      s" nested path wildcard, Such as: " +
+      s" 1. file://path/to/jar/*, file://path/to/jar/*/*" +
+      s" 2. hdfs://nameservice/path/to/jar/*, hdfs://nameservice/path/to/jar/*/*" +

Review comment:
       ditto, `\n`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-707471876


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34331/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704954497


   **[Test build #129505 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129505/testReport)** for PR 29881 at commit [`e3395f5`](https://github.com/apache/spark/commit/e3395f5e29ec0291b350dfa451479307fdc92c2c).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699583671


   **[Test build #129140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129140/testReport)** for PR 29881 at commit [`834c2a7`](https://github.com/apache/spark/commit/834c2a77a9a36e39f67eafd07def9b7e9f47639e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r502744398



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -88,12 +90,23 @@ private[spark] object HiveUtils extends Logging {
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format
+      |   support both local or remote paths.
+      | 4. A classpath in the standard format for both Hive and Hadoop, we should always
+      |   be fully qualified URL to indicate other file systems.
       """.stripMargin)
     .version("1.4.0")
     .stringConf
     .createWithDefault("builtin")
 
+  val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
+    .doc(s"Comma separated path of Hive jars, both support local and remote paths." +

Review comment:
       nit: `both support` -> `support both`.

##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -88,12 +90,23 @@ private[spark] object HiveUtils extends Logging {
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format
+      |   support both local or remote paths.
+      | 4. A classpath in the standard format for both Hive and Hadoop, we should always

Review comment:
       nit: `we should always` -> `it should always`?

##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +417,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
+            }
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil

Review comment:
       How are these jars be downloaded eventually? we'll need to call `FileSystem` API for that later right?

##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +417,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)

Review comment:
       Do we handle nested `*`, e.g., `path = `/foo/*/*`?

##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")
+          .flatMap {
+            case path if path.contains("\\") =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {
+                case null | "file" =>
+                  addLocalHiveJars(new File(uri.getPath))

Review comment:
       Should this comment on the option "path" instead of "classpath"?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704672346


   > I think we just need a new configuration that works like `SparkContext.addJar/addFile`, using `comma separator and resolving URIs with the HDFS's default scheme consistently if we should add this support.
   
   Like my first commit add a new configuration to provide a new way to config file path.  IMO, it's the best way to solve this conflict.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699584677


   **[Test build #129141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129141/testReport)** for PR 29881 at commit [`bf3c4b6`](https://github.com/apache/spark/commit/bf3c4b62f63f2f24945df38e487455004bd2d892).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704873573


   **[Test build #129505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129505/testReport)** for PR 29881 at commit [`e3395f5`](https://github.com/apache/spark/commit/e3395f5e29ec0291b350dfa451479307fdc92c2c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-709116134


   **[Test build #129832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129832/testReport)** for PR 29881 at commit [`b79ab0d`](https://github.com/apache/spark/commit/b79ab0d93a6e86fbf0706588d21c0540b3a4c0ab).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708857615


   **[Test build #129784 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129784/testReport)** for PR 29881 at commit [`b5241c8`](https://github.com/apache/spark/commit/b5241c810377a19d75cafb15c014a5cbe4fbccef).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708871976


   **[Test build #129789 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129789/testReport)** for PR 29881 at commit [`6fbe082`](https://github.com/apache/spark/commit/6fbe082416a778d1b71bf3c5560f8b9725938c6a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r496648397



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -397,27 +399,86 @@ private[spark] object HiveUtils extends Logging {
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
     } else {
-      // Convert to files and expand any directories.
-      val jars =
-        hiveMetastoreJars
-          .split(File.pathSeparator)
-          .flatMap {
-          case path if new File(path).getName == "*" =>
-            val files = new File(path).getParentFile.listFiles()
-            if (files == null) {
-              logWarning(s"Hive jar path '$path' does not exist.")
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
               Nil
             } else {
-              files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).toSeq
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
             }
-          case path =>
-            new File(path) :: Nil
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
         }
-          .map(_.toURI.toURL)
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        hiveMetastoreJars
+          .split(";")
+          .flatMap {
+            case path if path.contains("\\") =>

Review comment:
       `\\` can be in file names in other OSs. I don't think it makes sense to assume that this is a local path.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r506788022



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -80,20 +81,41 @@ private[spark] object HiveUtils extends Logging {
   val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
     .doc(s"""
       | Location of the jars that should be used to instantiate the HiveMetastoreClient.
-      | This property can be one of three options: "
+      | This property can be one of four options: "
       | 1. "builtin"
       |   Use Hive ${builtinHiveVersion}, which is bundled with the Spark assembly when
       |   <code>-Phive</code> is enabled. When this option is chosen,
       |   <code>spark.sql.hive.metastore.version</code> must be either
       |   <code>${builtinHiveVersion}</code> or not defined.
       | 2. "maven"
       |   Use Hive jars of specified version downloaded from Maven repositories.
-      | 3. A classpath in the standard format for both Hive and Hadoop.
+      | 3. "path"
+      |   Use Hive jars configured by `spark.sql.hive.metastore.jars.path` in comma separated format
+      |   support both local or remote paths, it should always be fully qualified URL to indicate
+      |   other file systems.
+      | 4. A classpath in the standard format for both Hive and Hadoop.
       """.stripMargin)
     .version("1.4.0")
     .stringConf
     .createWithDefault("builtin")
 
+  val HIVE_METASTORE_JARS_PATH = buildStaticConf("spark.sql.hive.metastore.jars.path")
+    .doc(s"Comma separated fully qualified URL of Hive jars, support both local and remote paths," +
+      s"Such as: " +
+      s" 1. file://path/to/jar/xxx.jar" +
+      s" 2. hdfs://nameservice/path/to/jar/xxx.jar" +
+      s" 3. /path/to/jar/ (path without URI scheme follow conf `fs.defaultFS`'s URI schema)" +
+      s" 4. [http/https/ftp]://path/to/jar/xxx.jar" +
+      s"For URI, we can't support path wildcard, but for other URL support nested path wildcard," +

Review comment:
       > can we be more specific? e.g. `http/https/ftp` doesn't support wildcard.
   
   Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-705953840


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34178/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710745392


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34538/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29881:
URL: https://github.com/apache/spark/pull/29881#discussion_r504668188



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -396,6 +419,93 @@ private[spark] object HiveUtils extends Logging {
         config = configurations,
         barrierPrefixes = hiveMetastoreBarrierPrefixes,
         sharedPrefixes = hiveMetastoreSharedPrefixes)
+    } else if (hiveMetastoreJars == "path") {
+
+      def addLocalHiveJars(file: File): Seq[URL] = {
+        if (file.getName == "*") {
+          val files = file.getParentFile.listFiles()
+          if (files == null) {
+            logWarning(s"Hive jar path '${file.getPath}' does not exist.")
+            Nil
+          } else {
+            files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURL).toSeq
+          }
+        } else {
+          file.toURL :: Nil
+        }
+      }
+
+      def checkRemoteHiveJars(path: String): Seq[URL] = {
+        try {
+          val hadoopPath = new Path(path)
+          val fs = hadoopPath.getFileSystem(hadoopConf)
+          if (hadoopPath.getName == "*") {
+            val parent = hadoopPath.getParent
+            if (!fs.exists(parent)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (!fs.getFileStatus(parent).isDirectory) {
+              logWarning(s"Hive Jar ${parent} is not a directory.")
+              Nil
+            } else {
+              fs.listStatus(parent).map(_.getPath.toUri.toURL)
+            }
+          } else {
+            if (!fs.exists(hadoopPath)) {
+              logWarning(s"Hive Jar ${path} does not exist.")
+              Nil
+            } else if (fs.getFileStatus(hadoopPath).isDirectory) {
+              logWarning(s"Hive Jar ${path} not allow directory without `*`")
+              Nil
+            } else {
+              // Since tar/tar.gz file we can't know it's final path yet, not support it
+              hadoopPath.toUri.toURL :: Nil
+            }
+          }
+        } catch {
+          case NonFatal(e) =>
+            logError(s"Failed to find $path to Hive Jars", e)
+            Nil
+        }
+      }
+
+      // Convert to files and expand any directories.
+      val jars =
+        HiveUtils.hiveMetastoreJarsPath(sqlConf)
+          .flatMap {
+            case path if Utils.isWindows =>
+              addLocalHiveJars(new File(path))
+            case path =>
+              val uri = new Path(path).toUri
+              uri.getScheme match {
+                case "file" | "local" =>

Review comment:
       > IIUC the Hadoop DFS API also supports local file systems.
   
   Not support local file system, but it can. support `http/https/ftp`
   ```
   ~/spark-3.1.0$ bin/spark-sql --master local --conf spark.sql.hive.metastore.jars=path --conf spark.sql.hive.metastore.jars.path=/home/hadoop/.ivy2/jars/*  --conf spark.sql.hive.metastore.version=2.3.7 --conf spark.driver.extraClassPath=$HIVE_CONF_DIR
   Setting default log level to "WARN".
   To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
   Exception in thread "main" org.apache.spark.sql.AnalysisException: Path does not exist: hdfs://redpoll/home/hadoop/.ivy2/jars/*;
   	at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$3(DataSource.scala:790)
   	at org.apache.spark.util.ThreadUtils$.$anonfun$parmap$2(ThreadUtils.scala:373)
   	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
   	at scala.util.Success.$anonfun$map$1(Try.scala:255)
   	at scala.util.Success.map(Try.scala:213)
   	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
   	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
   	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
   	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
   	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
   	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
   	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
   	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
   	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-712069175


   ping @cloud-fan 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-708671041


   **[Test build #129764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129764/testReport)** for PR 29881 at commit [`b5241c8`](https://github.com/apache/spark/commit/b5241c810377a19d75cafb15c014a5cbe4fbccef).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29881: [SPARK-32852][SQL} spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-699589036






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710739618


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-710745778


   **[Test build #129936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129936/testReport)** for PR 29881 at commit [`6e67c7b`](https://github.com/apache/spark/commit/6e67c7b9fcf9f5c66cb899f5f9000a5598492831).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29881: [SPARK-32852][SQL] spark.sql.hive.metastore.jars support HDFS location

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29881:
URL: https://github.com/apache/spark/pull/29881#issuecomment-704866420


   **[Test build #129504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129504/testReport)** for PR 29881 at commit [`c200d9b`](https://github.com/apache/spark/commit/c200d9b8503360cd0a268d1cb67e8c471df9705e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org