You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yaooqinn <gi...@git.apache.org> on 2017/11/06 05:33:17 UTC

[GitHub] spark pull request #19663: [SPARK-21888][Hive]add hadoop/hive/hdfs configura...

GitHub user yaooqinn opened a pull request:

    https://github.com/apache/spark/pull/19663

    [SPARK-21888][Hive]add hadoop/hive/hdfs configuration files in SPARK_CONF_DIR to distribute archive

    ## What changes were proposed in this pull request?
    When I ran self contained sql apps, such as
    ```scala
    import org.apache.spark.sql.SparkSession
    
    object ShowHiveTables {
      def main(args: Array[String]): Unit = {
        val spark = SparkSession
          .builder()
          .appName("Show Hive Tables")
          .enableHiveSupport()
          .getOrCreate()
        spark.sql("show tables").show()
        spark.stop()
      }
    }
    ```
    with **yarn cluster** mode and `hive-site.xml` correctly within `$SPARH_HOME/conf`,they failed to connect the right hive metestore for not seeing hive-site.xml in AM/Driver's classpath.
    
    Although submitting them with `--files/--jars local/path/to/hive-site.xml` or puting it to `$HADOOP_CONF_DIR/YARN_CONF_DIR` can make these apps works well in cluster mode as client mode, according to the official doc, see @ http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables
    > Configuration of Hive is done by placing your hive-site.xml, core-site.xml (for security configuration), and hdfs-site.xml (for HDFS configuration) file in conf/.
    
    We may respect these configuration files too or modify the doc for hive-tables in cluster mode.
    ## How was this patch tested?
    
    cc @cloud-fan @gatorsmile 


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/yaooqinn/spark SPARK-21888

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19663.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19663
    
----
commit 696dbd6e5b5af89752c3869264f70ceddb868baf
Author: Kent Yao <ya...@hotmail.com>
Date:   2017-11-06T04:56:04Z

    add hadoop/hive/hdfs configuration files in SPARK_CONF_DIR to distribute archive

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149017871
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -705,6 +705,19 @@ private[spark] class Client(
           }
         }
     
    +    val confDir =
    +      sys.env.getOrElse("SPARK_CONF_DIR", sys.env("SPARK_HOME") + File.separator + "conf")
    +    val dir = new File(confDir)
    +    if (dir.isDirectory) {
    +      val files = dir.listFiles(new FileFilter {
    +        override def accept(pathname: File): Boolean = {
    +          pathname.isFile && pathname.getName.endsWith("xml")
    --- End diff --
    
    I guess that we do not check the `$HADOOP(YARN)_CONF_DIR` either


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149016913
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -705,6 +705,19 @@ private[spark] class Client(
           }
         }
     
    +    val confDir =
    +      sys.env.getOrElse("SPARK_CONF_DIR", sys.env("SPARK_HOME") + File.separator + "conf")
    +    val dir = new File(confDir)
    +    if (dir.isDirectory) {
    +      val files = dir.listFiles(new FileFilter {
    +        override def accept(pathname: File): Boolean = {
    +          pathname.isFile && pathname.getName.endsWith("xml")
    --- End diff --
    
    According to the doc, 
    > Configuration of Hive is done by placing your hive-site.xml, core-site.xml (for security configuration), and hdfs-site.xml (for HDFS configuration) file in conf/.
    here we may not only get hive-site.xml


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-21888][SQL][Hive]add hadoop/hive/hbase/etc config...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    Please also add [YARN] tag to the PR title, this is actually a yarn problem.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    This is not SPARK-21888.  please file a separate jira for this issue. SPARK-21888 is meant to add things to the client classpath on the gateway/launcher box.
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149544716
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -687,6 +687,20 @@ private[spark] class Client(
       private def createConfArchive(): File = {
         val hadoopConfFiles = new HashMap[String, File]()
     
    +    // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
    +    val localConfDir = System.getProperty("SPARK_CONF_DIR",
    +      System.getProperty("SPARK_HOME") + File.separator + "conf")
    +    val dir = new File(localConfDir)
    +    if (dir.isDirectory) {
    +      val files = dir.listFiles(new FileFilter {
    +        override def accept(pathname: File): Boolean = {
    +          pathname.isFile && pathname.getName.endsWith("xml")
    --- End diff --
    
    `".xml"`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83576/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    **[Test build #83487 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83487/testReport)** for PR 19663 at commit [`e56f039`](https://github.com/apache/spark/commit/e56f0392a1781d143b5269bf7db949b6f0e1a884).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149544880
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -687,6 +687,20 @@ private[spark] class Client(
       private def createConfArchive(): File = {
         val hadoopConfFiles = new HashMap[String, File]()
     
    +    // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
    +    val localConfDir = System.getProperty("SPARK_CONF_DIR",
    +      System.getProperty("SPARK_HOME") + File.separator + "conf")
    +    val dir = new File(localConfDir)
    +    if (dir.isDirectory) {
    +      val files = dir.listFiles(new FileFilter {
    +        override def accept(pathname: File): Boolean = {
    +          pathname.isFile && pathname.getName.endsWith("xml")
    +        }
    +      })
    +      files.foreach { f => hadoopConfFiles(f.getName) = f }
    +    }
    +
    +    // Ensure HADOOP_CONF_DIR/YARN_CONF_DIR not overriding existing files
    --- End diff --
    
    This comment doesn't make a lot of sense, at least not in this position. What are you trying to say?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-21888][SQL][Hive]add hadoop/hive/hbase/etc...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149015858
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -705,6 +705,19 @@ private[spark] class Client(
           }
         }
     
    +    val confDir =
    +      sys.env.getOrElse("SPARK_CONF_DIR", sys.env("SPARK_HOME") + File.separator + "conf")
    +    val dir = new File(confDir)
    +    if (dir.isDirectory) {
    +      val files = dir.listFiles(new FileFilter {
    +        override def accept(pathname: File): Boolean = {
    +          pathname.isFile && pathname.getName.endsWith("xml")
    --- End diff --
    
    Shall we explicitly match the file name, like "hive-site.xml"? Looks like only check file name ends with "xml" will also include other unwanted files indefinitely.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    **[Test build #83529 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83529/testReport)** for PR 19663 at commit [`6e93d74`](https://github.com/apache/spark/commit/6e93d742633fe04ebab6d37095ac53bcaef1db98).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    **[Test build #83632 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83632/testReport)** for PR 19663 at commit [`61b342c`](https://github.com/apache/spark/commit/61b342c9d7e4145052c2d7edd835bd36f401087e).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83632/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149272398
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -705,6 +705,19 @@ private[spark] class Client(
           }
         }
     
    +    val confDir =
    +      sys.env.getOrElse("SPARK_CONF_DIR", sys.env("SPARK_HOME") + File.separator + "conf")
    +    val dir = new File(confDir)
    +    if (dir.isDirectory) {
    +      val files = dir.listFiles(new FileFilter {
    +        override def accept(pathname: File): Boolean = {
    +          pathname.isFile && pathname.getName.endsWith("xml")
    +        }
    +      })
    +      files.foreach { f => hadoopConfFiles(f.getName) = f }
    --- End diff --
    
    yes, we follow that order to build classpath. plz check https://github.com/apache/spark/blob/12ab7f7e89ec9e102859ab3b710815d3058a2e8d/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java#L134


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    **[Test build #83529 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83529/testReport)** for PR 19663 at commit [`6e93d74`](https://github.com/apache/spark/commit/6e93d742633fe04ebab6d37095ac53bcaef1db98).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83487/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    **[Test build #83487 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83487/testReport)** for PR 19663 at commit [`e56f039`](https://github.com/apache/spark/commit/e56f0392a1781d143b5269bf7db949b6f0e1a884).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    thanks, merging to master!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149066829
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -705,6 +705,19 @@ private[spark] class Client(
           }
         }
     
    +    val confDir =
    +      sys.env.getOrElse("SPARK_CONF_DIR", sys.env("SPARK_HOME") + File.separator + "conf")
    +    val dir = new File(confDir)
    +    if (dir.isDirectory) {
    +      val files = dir.listFiles(new FileFilter {
    +        override def accept(pathname: File): Boolean = {
    +          pathname.isFile && pathname.getName.endsWith("xml")
    +        }
    +      })
    +      files.foreach { f => hadoopConfFiles(f.getName) = f }
    --- End diff --
    
    This indicates files in SPARK_CONF_DIR have higher priority than HADOOP_CONF_DIR or `YARN_CONF_DIR`, is it expected?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149544676
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -687,6 +687,20 @@ private[spark] class Client(
       private def createConfArchive(): File = {
         val hadoopConfFiles = new HashMap[String, File]()
     
    +    // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
    +    val localConfDir = System.getProperty("SPARK_CONF_DIR",
    --- End diff --
    
    `SPARK_CONF_DIR` is set by Spark's launch scripts, so you should just be able to do:
    
    ```
    sys.env.get("SPARK_CONF_DIR").foreach { ... }
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149017279
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -705,6 +705,19 @@ private[spark] class Client(
           }
         }
     
    +    val confDir =
    +      sys.env.getOrElse("SPARK_CONF_DIR", sys.env("SPARK_HOME") + File.separator + "conf")
    +    val dir = new File(confDir)
    +    if (dir.isDirectory) {
    +      val files = dir.listFiles(new FileFilter {
    +        override def accept(pathname: File): Boolean = {
    +          pathname.isFile && pathname.getName.endsWith("xml")
    --- End diff --
    
    Yes, I understand. My question is that do we need to explicitly check the expected file names, rather than blindly match any xml file?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/19663


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-21888][Hive]add hadoop/hive/hdfs configuration fi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83529/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149849250
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -687,6 +687,20 @@ private[spark] class Client(
       private def createConfArchive(): File = {
         val hadoopConfFiles = new HashMap[String, File]()
     
    +    // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
    +    val localConfDir = System.getProperty("SPARK_CONF_DIR",
    --- End diff --
    
    ok, thanks for you advise


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    **[Test build #83576 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83576/testReport)** for PR 19663 at commit [`f8c1f63`](https://github.com/apache/spark/commit/f8c1f63944c602a00802356f94788464320ffa3f).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149561877
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -687,6 +687,20 @@ private[spark] class Client(
       private def createConfArchive(): File = {
         val hadoopConfFiles = new HashMap[String, File]()
     
    +    // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
    +    val localConfDir = System.getProperty("SPARK_CONF_DIR",
    --- End diff --
    
    not exactly till now  , plz check https://github.com/apache/spark/pull/19688


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149561925
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -687,6 +687,20 @@ private[spark] class Client(
       private def createConfArchive(): File = {
         val hadoopConfFiles = new HashMap[String, File]()
     
    +    // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
    +    val localConfDir = System.getProperty("SPARK_CONF_DIR",
    +      System.getProperty("SPARK_HOME") + File.separator + "conf")
    +    val dir = new File(localConfDir)
    +    if (dir.isDirectory) {
    +      val files = dir.listFiles(new FileFilter {
    +        override def accept(pathname: File): Boolean = {
    +          pathname.isFile && pathname.getName.endsWith("xml")
    +        }
    +      })
    +      files.foreach { f => hadoopConfFiles(f.getName) = f }
    +    }
    +
    +    // Ensure HADOOP_CONF_DIR/YARN_CONF_DIR not overriding existing files
    --- End diff --
    
    ok, i'd remove it


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    ok to test


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    **[Test build #83576 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83576/testReport)** for PR 19663 at commit [`f8c1f63`](https://github.com/apache/spark/commit/f8c1f63944c602a00802356f94788464320ffa3f).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149751233
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -687,6 +687,20 @@ private[spark] class Client(
       private def createConfArchive(): File = {
         val hadoopConfFiles = new HashMap[String, File]()
     
    +    // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
    +    val localConfDir = System.getProperty("SPARK_CONF_DIR",
    --- End diff --
    
    So it's being fixed and you can apply my suggestion, no?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...

Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19663#discussion_r149561888
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -687,6 +687,20 @@ private[spark] class Client(
       private def createConfArchive(): File = {
         val hadoopConfFiles = new HashMap[String, File]()
     
    +    // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
    +    val localConfDir = System.getProperty("SPARK_CONF_DIR",
    +      System.getProperty("SPARK_HOME") + File.separator + "conf")
    +    val dir = new File(localConfDir)
    +    if (dir.isDirectory) {
    +      val files = dir.listFiles(new FileFilter {
    +        override def accept(pathname: File): Boolean = {
    +          pathname.isFile && pathname.getName.endsWith("xml")
    --- End diff --
    
    ok


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19663
  
    **[Test build #83632 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83632/testReport)** for PR 19663 at commit [`61b342c`](https://github.com/apache/spark/commit/61b342c9d7e4145052c2d7edd835bd36f401087e).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org