You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yaooqinn <gi...@git.apache.org> on 2017/11/06 05:33:17 UTC
[GitHub] spark pull request #19663: [SPARK-21888][Hive]add hadoop/hive/hdfs configura...
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/19663
[SPARK-21888][Hive]add hadoop/hive/hdfs configuration files in SPARK_CONF_DIR to distribute archive
## What changes were proposed in this pull request?
When I ran self contained sql apps, such as
```scala
import org.apache.spark.sql.SparkSession
object ShowHiveTables {
def main(args: Array[String]): Unit = {
val spark = SparkSession
.builder()
.appName("Show Hive Tables")
.enableHiveSupport()
.getOrCreate()
spark.sql("show tables").show()
spark.stop()
}
}
```
with **yarn cluster** mode and `hive-site.xml` correctly within `$SPARH_HOME/conf`,they failed to connect the right hive metestore for not seeing hive-site.xml in AM/Driver's classpath.
Although submitting them with `--files/--jars local/path/to/hive-site.xml` or puting it to `$HADOOP_CONF_DIR/YARN_CONF_DIR` can make these apps works well in cluster mode as client mode, according to the official doc, see @ http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables
> Configuration of Hive is done by placing your hive-site.xml, core-site.xml (for security configuration), and hdfs-site.xml (for HDFS configuration) file in conf/.
We may respect these configuration files too or modify the doc for hive-tables in cluster mode.
## How was this patch tested?
cc @cloud-fan @gatorsmile
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yaooqinn/spark SPARK-21888
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19663.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19663
----
commit 696dbd6e5b5af89752c3869264f70ceddb868baf
Author: Kent Yao <ya...@hotmail.com>
Date: 2017-11-06T04:56:04Z
add hadoop/hive/hdfs configuration files in SPARK_CONF_DIR to distribute archive
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149017871
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -705,6 +705,19 @@ private[spark] class Client(
}
}
+ val confDir =
+ sys.env.getOrElse("SPARK_CONF_DIR", sys.env("SPARK_HOME") + File.separator + "conf")
+ val dir = new File(confDir)
+ if (dir.isDirectory) {
+ val files = dir.listFiles(new FileFilter {
+ override def accept(pathname: File): Boolean = {
+ pathname.isFile && pathname.getName.endsWith("xml")
--- End diff --
I guess that we do not check the `$HADOOP(YARN)_CONF_DIR` either
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149016913
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -705,6 +705,19 @@ private[spark] class Client(
}
}
+ val confDir =
+ sys.env.getOrElse("SPARK_CONF_DIR", sys.env("SPARK_HOME") + File.separator + "conf")
+ val dir = new File(confDir)
+ if (dir.isDirectory) {
+ val files = dir.listFiles(new FileFilter {
+ override def accept(pathname: File): Boolean = {
+ pathname.isFile && pathname.getName.endsWith("xml")
--- End diff --
According to the doc,
> Configuration of Hive is done by placing your hive-site.xml, core-site.xml (for security configuration), and hdfs-site.xml (for HDFS configuration) file in conf/.
here we may not only get hive-site.xml
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-21888][SQL][Hive]add hadoop/hive/hbase/etc config...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19663
Please also add [YARN] tag to the PR title, this is actually a yarn problem.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/19663
This is not SPARK-21888. please file a separate jira for this issue. SPARK-21888 is meant to add things to the client classpath on the gateway/launcher box.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19663
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149544716
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -687,6 +687,20 @@ private[spark] class Client(
private def createConfArchive(): File = {
val hadoopConfFiles = new HashMap[String, File]()
+ // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
+ val localConfDir = System.getProperty("SPARK_CONF_DIR",
+ System.getProperty("SPARK_HOME") + File.separator + "conf")
+ val dir = new File(localConfDir)
+ if (dir.isDirectory) {
+ val files = dir.listFiles(new FileFilter {
+ override def accept(pathname: File): Boolean = {
+ pathname.isFile && pathname.getName.endsWith("xml")
--- End diff --
`".xml"`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19663
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83576/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19663
**[Test build #83487 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83487/testReport)** for PR 19663 at commit [`e56f039`](https://github.com/apache/spark/commit/e56f0392a1781d143b5269bf7db949b6f0e1a884).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19663
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149544880
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -687,6 +687,20 @@ private[spark] class Client(
private def createConfArchive(): File = {
val hadoopConfFiles = new HashMap[String, File]()
+ // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
+ val localConfDir = System.getProperty("SPARK_CONF_DIR",
+ System.getProperty("SPARK_HOME") + File.separator + "conf")
+ val dir = new File(localConfDir)
+ if (dir.isDirectory) {
+ val files = dir.listFiles(new FileFilter {
+ override def accept(pathname: File): Boolean = {
+ pathname.isFile && pathname.getName.endsWith("xml")
+ }
+ })
+ files.foreach { f => hadoopConfFiles(f.getName) = f }
+ }
+
+ // Ensure HADOOP_CONF_DIR/YARN_CONF_DIR not overriding existing files
--- End diff --
This comment doesn't make a lot of sense, at least not in this position. What are you trying to say?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-21888][SQL][Hive]add hadoop/hive/hbase/etc...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149015858
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -705,6 +705,19 @@ private[spark] class Client(
}
}
+ val confDir =
+ sys.env.getOrElse("SPARK_CONF_DIR", sys.env("SPARK_HOME") + File.separator + "conf")
+ val dir = new File(confDir)
+ if (dir.isDirectory) {
+ val files = dir.listFiles(new FileFilter {
+ override def accept(pathname: File): Boolean = {
+ pathname.isFile && pathname.getName.endsWith("xml")
--- End diff --
Shall we explicitly match the file name, like "hive-site.xml"? Looks like only check file name ends with "xml" will also include other unwanted files indefinitely.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19663
**[Test build #83529 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83529/testReport)** for PR 19663 at commit [`6e93d74`](https://github.com/apache/spark/commit/6e93d742633fe04ebab6d37095ac53bcaef1db98).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19663
**[Test build #83632 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83632/testReport)** for PR 19663 at commit [`61b342c`](https://github.com/apache/spark/commit/61b342c9d7e4145052c2d7edd835bd36f401087e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19663
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83632/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19663
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149272398
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -705,6 +705,19 @@ private[spark] class Client(
}
}
+ val confDir =
+ sys.env.getOrElse("SPARK_CONF_DIR", sys.env("SPARK_HOME") + File.separator + "conf")
+ val dir = new File(confDir)
+ if (dir.isDirectory) {
+ val files = dir.listFiles(new FileFilter {
+ override def accept(pathname: File): Boolean = {
+ pathname.isFile && pathname.getName.endsWith("xml")
+ }
+ })
+ files.foreach { f => hadoopConfFiles(f.getName) = f }
--- End diff --
yes, we follow that order to build classpath. plz check https://github.com/apache/spark/blob/12ab7f7e89ec9e102859ab3b710815d3058a2e8d/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java#L134
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19663
**[Test build #83529 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83529/testReport)** for PR 19663 at commit [`6e93d74`](https://github.com/apache/spark/commit/6e93d742633fe04ebab6d37095ac53bcaef1db98).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19663
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83487/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19663
**[Test build #83487 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83487/testReport)** for PR 19663 at commit [`e56f039`](https://github.com/apache/spark/commit/e56f0392a1781d143b5269bf7db949b6f0e1a884).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19663
thanks, merging to master!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149066829
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -705,6 +705,19 @@ private[spark] class Client(
}
}
+ val confDir =
+ sys.env.getOrElse("SPARK_CONF_DIR", sys.env("SPARK_HOME") + File.separator + "conf")
+ val dir = new File(confDir)
+ if (dir.isDirectory) {
+ val files = dir.listFiles(new FileFilter {
+ override def accept(pathname: File): Boolean = {
+ pathname.isFile && pathname.getName.endsWith("xml")
+ }
+ })
+ files.foreach { f => hadoopConfFiles(f.getName) = f }
--- End diff --
This indicates files in SPARK_CONF_DIR have higher priority than HADOOP_CONF_DIR or `YARN_CONF_DIR`, is it expected?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149544676
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -687,6 +687,20 @@ private[spark] class Client(
private def createConfArchive(): File = {
val hadoopConfFiles = new HashMap[String, File]()
+ // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
+ val localConfDir = System.getProperty("SPARK_CONF_DIR",
--- End diff --
`SPARK_CONF_DIR` is set by Spark's launch scripts, so you should just be able to do:
```
sys.env.get("SPARK_CONF_DIR").foreach { ... }
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149017279
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -705,6 +705,19 @@ private[spark] class Client(
}
}
+ val confDir =
+ sys.env.getOrElse("SPARK_CONF_DIR", sys.env("SPARK_HOME") + File.separator + "conf")
+ val dir = new File(confDir)
+ if (dir.isDirectory) {
+ val files = dir.listFiles(new FileFilter {
+ override def accept(pathname: File): Boolean = {
+ pathname.isFile && pathname.getName.endsWith("xml")
--- End diff --
Yes, I understand. My question is that do we need to explicitly check the expected file names, rather than blindly match any xml file?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19663
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-21888][Hive]add hadoop/hive/hdfs configuration fi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19663
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19663
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83529/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149849250
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -687,6 +687,20 @@ private[spark] class Client(
private def createConfArchive(): File = {
val hadoopConfFiles = new HashMap[String, File]()
+ // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
+ val localConfDir = System.getProperty("SPARK_CONF_DIR",
--- End diff --
ok, thanks for you advise
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19663
**[Test build #83576 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83576/testReport)** for PR 19663 at commit [`f8c1f63`](https://github.com/apache/spark/commit/f8c1f63944c602a00802356f94788464320ffa3f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19663
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149561877
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -687,6 +687,20 @@ private[spark] class Client(
private def createConfArchive(): File = {
val hadoopConfFiles = new HashMap[String, File]()
+ // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
+ val localConfDir = System.getProperty("SPARK_CONF_DIR",
--- End diff --
not exactly till now , plz check https://github.com/apache/spark/pull/19688
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149561925
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -687,6 +687,20 @@ private[spark] class Client(
private def createConfArchive(): File = {
val hadoopConfFiles = new HashMap[String, File]()
+ // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
+ val localConfDir = System.getProperty("SPARK_CONF_DIR",
+ System.getProperty("SPARK_HOME") + File.separator + "conf")
+ val dir = new File(localConfDir)
+ if (dir.isDirectory) {
+ val files = dir.listFiles(new FileFilter {
+ override def accept(pathname: File): Boolean = {
+ pathname.isFile && pathname.getName.endsWith("xml")
+ }
+ })
+ files.foreach { f => hadoopConfFiles(f.getName) = f }
+ }
+
+ // Ensure HADOOP_CONF_DIR/YARN_CONF_DIR not overriding existing files
--- End diff --
ok, i'd remove it
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-21888][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19663
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19663
**[Test build #83576 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83576/testReport)** for PR 19663 at commit [`f8c1f63`](https://github.com/apache/spark/commit/f8c1f63944c602a00802356f94788464320ffa3f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149751233
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -687,6 +687,20 @@ private[spark] class Client(
private def createConfArchive(): File = {
val hadoopConfFiles = new HashMap[String, File]()
+ // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
+ val localConfDir = System.getProperty("SPARK_CONF_DIR",
--- End diff --
So it's being fixed and you can apply my suggestion, no?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hba...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19663#discussion_r149561888
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -687,6 +687,20 @@ private[spark] class Client(
private def createConfArchive(): File = {
val hadoopConfFiles = new HashMap[String, File]()
+ // SPARK_CONF_DIR shows up in the classpath before HADOOP_CONF_DIR/YARN_CONF_DIR
+ val localConfDir = System.getProperty("SPARK_CONF_DIR",
+ System.getProperty("SPARK_HOME") + File.separator + "conf")
+ val dir = new File(localConfDir)
+ if (dir.isDirectory) {
+ val files = dir.listFiles(new FileFilter {
+ override def accept(pathname: File): Boolean = {
+ pathname.isFile && pathname.getName.endsWith("xml")
--- End diff --
ok
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19663: [SPARK-22463][YARN][SQL][Hive]add hadoop/hive/hbase/etc ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19663
**[Test build #83632 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83632/testReport)** for PR 19663 at commit [`61b342c`](https://github.com/apache/spark/commit/61b342c9d7e4145052c2d7edd835bd36f401087e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org