You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sarutak <gi...@git.apache.org> on 2016/06/17 15:56:33 UTC
[GitHub] spark pull request #13738: [SPARK-11227][CORE] Jobs can throw UnknownHostExc...
GitHub user sarutak opened a pull request:
https://github.com/apache/spark/pull/13738
[SPARK-11227][CORE] Jobs can throw UnknownHostException when NameNode HA is enabled.
## What changes were proposed in this pull request?
If the following conditions are satisfied, executors don't load properties in `hdfs-site.xml` and UnknownHostException can be thrown.
(1) NameNode HA is enabled
(2) spark.eventLogging is enabled or logging path is NOT on HDFS
(3) Using Standalone or Mesos for the cluster manager
(4) There are no code to load `HdfsCondition` class in the driver regardless of directly or indirectly.
(5) The tasks access to HDFS
(There might be some more conditions...)
For example, following code causes UnknownHostException when the conditions above are satisfied.
```
sc.textFile("<path on HDFS>").collect
```
```
java.lang.IllegalArgumentException: java.net.UnknownHostException: hacluster
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:656)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:438)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:411)
at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$32.apply(SparkContext.scala:986)
at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$32.apply(SparkContext.scala:986)
at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:177)
at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:177)
at scala.Option.map(Option.scala:146)
at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:177)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:213)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:209)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:102)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: hacluster
```
But following code doesn't cause the Exception because `textFile` method load `HdfsConfiguration` indirectly.
```
sc.textFile("<path on HDFS>").collect
```
When a job includes some operations which access to HDFS, the object of `org.apache.hadoop.Configuration` is wrapped by `SerializableConfiguration`, serialized and broadcasted from driver to executors and each executor deserialize the object with `loadDefaults` false so HDFS related properties should be set before broadcasted.
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
Tested manually on my standalone cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sarutak/spark SPARK-11227
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13738.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13738
----
commit d41bfe09f1a32d6fff3d084b8f5452f615397744
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Date: 2016-06-17T15:27:22Z
Force to load HdfsConfiguration
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13738
**[Test build #62819 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62819/consoleFull)** for PR 13738 at commit [`fca2b4a`](https://github.com/apache/spark/commit/fca2b4a319e79e14f9c8975d337b47fdaeac8e5e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
you should file a bug with databricks to fix spark-csv. sorry haven't had a chance to get back to this, I'll look more today.
I would prefer to only load it when you are doing something hdfs related which is why I was mentioning the specific routines. Just need to look at how to do it for the ones that don't take a file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by soldiershen <gi...@git.apache.org>.
Github user soldiershen commented on the issue:
https://github.com/apache/spark/pull/13738
Hi guys,I met same issue(spark 1.6.0) when using "val rdd1 = hiveContext.sql("select SUBSTR(startjointime,1,10)... println(rdd1.collect().toString)"
but it's normal when using hiveContext.sql("show tables").collect().foreach(println)
For now,is there any solution to make it normal
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13738: [SPARK-11227][CORE] UnknownHostException can be t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13738#discussion_r67813873
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala ---
@@ -421,6 +421,13 @@ object SparkHadoopUtil {
val SPARK_YARN_CREDS_COUNTER_DELIM = "-"
+ // Just load HdfsConfiguration into the class loader to add
+ // hdfs-site.xml as a default configuration file otherwise
+ // some HDFS related configurations doesn't ship to Executors and
+ // it can cause UnknownHostException when NameNode HA is enabled.
+ // See SPARK-11227 for more details.
+ Utils.classForName("org.apache.hadoop.hdfs.HdfsConfiguration")
--- End diff --
Just reference it in any way, but, I guess we should ask, what does classloading do that we need, and is there any way to do that directly? this is fairly indirect. Is it that ` Configuration.addDefaultResource("hdfs-site.xml");` must be called?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13738
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
the jira says this worked in spark 1.4 but not spark 1.5 or 1.6, do we know why? was spark loading the configuration somewhere else early on in spark 1.4?
I must be misunderstanding your description because it says the following code causes an exception: sc.textFile("<path on HDFS>").collect but then below the exception you say it (sc.textFile("<path on HDFS>").collect) doesn't cause an exception because it loads the HdfsConfiguration object.
What exact code caused the exception?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
+1, thanks @sarutak
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13738
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
Sorry for the late response. I'll try it the way you mentioned.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
@soldiershen Doesn't this patch address the issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
thanks for the explanation, I understand now.
ok on the loading it looks like https://issues.apache.org/jira/browse/SPARK-8135 broke this because its not loading the defaults anymore. The reason it stopped is because that is expensive.
I need to look a bit more to see if better way that doesn't use private interface.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
@tgravescs I reproduced this with following condition.
(1) Made `spark-default.conf` empty
(2) Only `HADOOP_CONF_DIR=/path/to/hadoop-conf` in spark-env.sh
(3) NameNode HA is enabled and settings are in hdfs-site.xml in the client where I ran spark-submit.
(4) Using standalone cluster
(5) Submitted job by following command
```
spark-submit \
--master spark://host:port \
--deploy-mode client \
--class ReproduceApp \
reproduceapp.jar \
<input_path_to_hdfs>
```
Following code is used to reproduce this.
```
import org.apache.spark._
object ReproduceApp {
def main(args: Array[String]) {
val conf = new SparkConf()
val sc = new SparkContext(conf)
val fileName = args(0)
sc.textFile(fileName).collect
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
does this happen when you use the dataframe load method? I'm guessing not because the datasource code does a hdfsPath.getFileSystem.
for hadoopFile, textFile, newApiHadoopFile since we have the path I think we could just add in a FileSystem.get(path, conf) on them and if its an hdfs path it would call HdfsConfiguration underneath. The hard part is the hadoopRDD, newApiHadoopRDD since we don't have the path passed directly, its set via a conf. Need to look at that one more.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by soldiershen <gi...@git.apache.org>.
Github user soldiershen commented on the issue:
https://github.com/apache/spark/pull/13738
@sarutak got it. I add hdfs conf file to specific the host .Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
@soldiershen did you try this patch rather then putting the hdfs conf file on the hosts?
@sarutak sorry been out on vacation and going on vacation again, probably won't get to this til next week.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13738: [SPARK-11227][CORE] UnknownHostException can be t...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13738#discussion_r67810196
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala ---
@@ -421,6 +421,13 @@ object SparkHadoopUtil {
val SPARK_YARN_CREDS_COUNTER_DELIM = "-"
+ // Just load HdfsConfiguration into the class loader to add
+ // hdfs-site.xml as a default configuration file otherwise
+ // some HDFS related configurations doesn't ship to Executors and
+ // it can cause UnknownHostException when NameNode HA is enabled.
+ // See SPARK-11227 for more details.
+ Utils.classForName("org.apache.hadoop.hdfs.HdfsConfiguration")
--- End diff --
Does this need to be loaded by reflection -- is it not in Hadoop 2.2?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13738: [SPARK-11227][CORE] UnknownHostException can be t...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/13738#discussion_r67813110
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala ---
@@ -421,6 +421,13 @@ object SparkHadoopUtil {
val SPARK_YARN_CREDS_COUNTER_DELIM = "-"
+ // Just load HdfsConfiguration into the class loader to add
+ // hdfs-site.xml as a default configuration file otherwise
+ // some HDFS related configurations doesn't ship to Executors and
+ // it can cause UnknownHostException when NameNode HA is enabled.
+ // See SPARK-11227 for more details.
+ Utils.classForName("org.apache.hadoop.hdfs.HdfsConfiguration")
--- End diff --
As you mentioned, Hadoop 2.2 have `HdfsConfiguration` but are there any way to load `HdfsConfiguration` explicitly without reflection?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
@tgravescs I didn't debug against 1.4 but I think the reason why 1.5+ can throw UnknownHostException while 1.4 works successfully is related to the difference between `SerializableWritable` and `SerializableConfiguration`. The former allow executor to load hdfs-site.xml even if driver doesn't load it but the latter disallow executor to load it. In 1.4, `SerializableWritable` is used and in 1.5+, `SerializableConfiguration` is used in `HadoopRDD`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13738
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60711/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13738: [SPARK-11227][CORE] UnknownHostException can be t...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/13738#discussion_r67815037
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala ---
@@ -421,6 +421,13 @@ object SparkHadoopUtil {
val SPARK_YARN_CREDS_COUNTER_DELIM = "-"
+ // Just load HdfsConfiguration into the class loader to add
+ // hdfs-site.xml as a default configuration file otherwise
+ // some HDFS related configurations doesn't ship to Executors and
+ // it can cause UnknownHostException when NameNode HA is enabled.
+ // See SPARK-11227 for more details.
+ Utils.classForName("org.apache.hadoop.hdfs.HdfsConfiguration")
--- End diff --
Yeah, `Configuration.addDefaultResource("hdfs-site.xml")` may be better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
@tgravescs No problem. Have a nice vacation :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
In Spark 2.0, this issue cannot happen when we use the dataframe load method because as you mentioned, all of file-based datasource does a `hdfsPath.getFileSystem`.
I noticed there is at least one case when this issue can happen. It's a corner case.
When we use Spark 1.6 and `spark-csv` (https://github.com/databricks/spark-csv), this issue can happen.
I was able to reproduce this by following code with Spark 1.6.1 and spark-csv 1.4.0.
```
import org.apache.spark._
import org.apache.spark.sql._
object ReproduceApp2 {
def main(args: Array[String]) {
val conf = new SparkConf()
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
val input = args(0)
val output = args(1)
sqlContext
.read.format("csv")
.option("header", "true")
.load(input)
.write.format("json")
.mode(SaveMode.Overwrite)
.save(output)
}
}
```
This is because `DefaultSource` of `spark-csv` does not implement `HadoopFsRelationProvider`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
and on the second question, what command was used to reproduce this exactly?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
Sorry for my delay on this, so it looks like if we create any FileSystem is will cause the service loaders to load all services, so in hadoopRDD, newApiHadoopRDD where we don't know the path we could just call getLocal to get the local filesystem and it will load the hdfs ones. Its a bit ugly since we are relying on them not changing that behavior but the best I've come up with.
All the other calls like I mention above can call FileSystem.get(path, conf).
Could you try that out and see if that works?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13738: [SPARK-11227][CORE] UnknownHostException can be t...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13738#discussion_r67876187
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala ---
@@ -421,6 +421,13 @@ object SparkHadoopUtil {
val SPARK_YARN_CREDS_COUNTER_DELIM = "-"
+ // Just load HdfsConfiguration into the class loader to add
+ // hdfs-site.xml as a default configuration file otherwise
+ // some HDFS related configurations doesn't ship to Executors and
+ // it can cause UnknownHostException when NameNode HA is enabled.
+ // See SPARK-11227 for more details.
+ Utils.classForName("org.apache.hadoop.hdfs.HdfsConfiguration")
--- End diff --
actually I'd prefer not to reference hdfs-site.xml directly. HdfsConfiguration should know what it needs to load. For instance it also loads the defaults. HdfsConfiguration is also marked as @InterfaceAudience.Private so ideally we shouldn't be using it directly. Based on my other comments I would like to understand better why this isn't loaded on driver already.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
Maybe anyone of @rxin , @srowen or @tgravescs could review this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] Jobs can throw UnknownHostException ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13738
**[Test build #60711 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60711/consoleFull)** for PR 13738 at commit [`d41bfe0`](https://github.com/apache/spark/commit/d41bfe09f1a32d6fff3d084b8f5452f615397744).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13738
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62819/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13738: [SPARK-11227][CORE] UnknownHostException can be t...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13738
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13738
**[Test build #60711 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60711/consoleFull)** for PR 13738 at commit [`d41bfe0`](https://github.com/apache/spark/commit/d41bfe09f1a32d6fff3d084b8f5452f615397744).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13738
**[Test build #62819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62819/consoleFull)** for PR 13738 at commit [`fca2b4a`](https://github.com/apache/spark/commit/fca2b4a319e79e14f9c8975d337b47fdaeac8e5e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13738: [SPARK-11227][CORE] UnknownHostException can be thrown w...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
Yeah, I agree that `spark-csv` case is the special one and not a matter of Spark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org