Strange exception while accessing hdfs from spark.

[Julien Carme <ju...@gmail.com>]

Hello,

I have been using Spark for quite some time, and I now get this error
(please stderr output below) when accessing hdfs. It seems to come from
Hadoop, however, I can access hdfs from the command line without any
problem.

The WARN on the first seems to be key, because it never appeared
previously. My HADOOP_CONF_DIR is correctly set, so I don't know why
libhadoop cannot be loaded.

Please note that this " java.net.UnknownHostException: crm" is rather
strange because  crm is not an host, it is the name of my hadoop cluster,
as specified in hadoop core.xml file in dfs.nameservices and fs.defaultFS
hadoop properties.

Thanks for your help.


14/09/17 22:33:49 WARN BlockReaderLocal: The short-circuit local reads
feature cannot
 be used because libhadoop cannot be loaded.
14/09/17 22:33:49 WARN BlockReaderLocal: The short-circuit local reads
feature cannot
 be used because libhadoop cannot be loaded.
14/09/17 22:34:01 WARN TaskSetManager: Lost TID 20 (task 1.0:20)
14/09/17 22:34:01 WARN TaskSetManager: Loss was due to
java.lang.IllegalArgumentExcep
tion
java.lang.IllegalArgumentException: java.net.UnknownHostException: crm
        at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.jav
a:418)
        at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.ja
va:231)
        at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:13
9)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:510)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453)
        at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSys
tem.java:136)
        at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2433)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
        at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:166)
        at
org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:653)
        at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.jav
a:389)
        at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.jav
a:362)
        at
org.apache.spark.SparkContext$$anonfun$22.apply(SparkContext.scala:546)
        at
org.apache.spark.SparkContext$$anonfun$22.apply(SparkContext.scala:546)
        at
org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$1.apply(HadoopRDD.scala