You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by mohankreddy <mr...@beanatomics.com> on 2014/02/10 09:44:31 UTC
EOF Exception when trying to access hdfs://
I am getting the following error when trying to access my data using hdfs://
....... Not sure how to fix this one.
" java.io.IOException: Call to server1/10.85.85.17:9000 failed on local
exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1107)
at org.apache.hadoop.ipc.Client.call(Client.java:1075)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy8.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:176)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:140)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
at org.apache.spark.mllib.recommendation.ALS.run(ALS.scala:139)
at org.apache.spark.mllib.recommendation.ALS$.main(ALS.scala:594)
at org.apache.spark.mllib.recommendation.ALS.main(ALS.scala)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.ja
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/EOF-Exception-when-trying-to-access-hdfs-tp1347.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: EOF Exception when trying to access hdfs://
Posted by mohankreddy <mr...@beanatomics.com>.
Thanks, I resolved the issue. There was a firewall rule which was preventing
from accessing few ports.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/EOF-Exception-when-trying-to-access-hdfs-tp1347p1419.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: EOF Exception when trying to access hdfs://
Posted by Amit Behera <am...@gmail.com>.
Hi Mohan,
So try with adding the following dependency in your sbt file or maven
pom.xml file and rebuild your application and then run it.
SBT:
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.0.0-cdh4.4.0"
// If using CDH, also add Cloudera repo
resolvers += "Cloudera Repository" at "
https://repository.cloudera.com/artifactory/cloudera-repos/"
Maven:
<project>
<dependencies>
...
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.0-cdh4.4.0</version>
</dependency>
</dependencies>
<!-- If using CDH, also add Cloudera repo -->
<repositories>
...
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories></project>
On Tue, Feb 11, 2014 at 2:31 AM, mohankreddy <mr...@beanatomics.com> wrote:
> Hadoop version : 2.0.0-cdh4.4.0
> Spark: 0.9
>
> thanks
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/EOF-Exception-when-trying-to-access-hdfs-tp1347p1370.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: EOF Exception when trying to access hdfs://
Posted by mohankreddy <mr...@beanatomics.com>.
Hadoop version : 2.0.0-cdh4.4.0
Spark: 0.9
thanks
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/EOF-Exception-when-trying-to-access-hdfs-tp1347p1370.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: EOF Exception when trying to access hdfs://
Posted by Amit Behera <am...@gmail.com>.
hi Mohan,
could you please tell me the hadoop version and the spark version
on which you are working on.
On Mon, Feb 10, 2014 at 3:37 PM, Amit Behera <am...@gmail.com> wrote:
> Please go to hadoop configuration directory and open core-site.xml and
> check the IP and port for HDFS, mentioned on the value of "fs.default.name"
> . After that specify the same IP and Port number on your code.
> format *hdfs://<ip>:port/*
>
> I hope it will work.....
>
>
> On Mon, Feb 10, 2014 at 2:14 PM, mohankreddy <mr...@beanatomics.com>wrote:
>
>> I am getting the following error when trying to access my data using
>> hdfs://
>> ....... Not sure how to fix this one.
>>
>> " java.io.IOException: Call to server1/10.85.85.17:9000 failed on local
>> exception: java.io.EOFException
>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1107)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1075)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>> at $Proxy8.getProtocolVersion(Unknown Source)
>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
>> at
>> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
>> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
>> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
>> at
>>
>> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
>> at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
>> at
>>
>> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:176)
>> at
>>
>> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
>> at
>> org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:140)
>> at
>> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
>> at
>> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
>> at scala.Option.getOrElse(Option.scala:120)
>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
>> at
>> org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
>> at
>> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
>> at
>> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
>> at scala.Option.getOrElse(Option.scala:120)
>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
>> at
>> org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
>> at
>> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
>> at
>> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
>> at scala.Option.getOrElse(Option.scala:120)
>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
>> at org.apache.spark.mllib.recommendation.ALS.run(ALS.scala:139)
>> at org.apache.spark.mllib.recommendation.ALS$.main(ALS.scala:594)
>> at org.apache.spark.mllib.recommendation.ALS.main(ALS.scala)
>> Caused by: java.io.EOFException
>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>> at
>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.ja
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/EOF-Exception-when-trying-to-access-hdfs-tp1347.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>
Re: EOF Exception when trying to access hdfs://
Posted by Amit Behera <am...@gmail.com>.
Please go to hadoop configuration directory and open core-site.xml and
check the IP and port for HDFS, mentioned on the value of "fs.default.name"
. After that specify the same IP and Port number on your code.
format *hdfs://<ip>:port/*
I hope it will work.....
On Mon, Feb 10, 2014 at 2:14 PM, mohankreddy <mr...@beanatomics.com> wrote:
> I am getting the following error when trying to access my data using
> hdfs://
> ....... Not sure how to fix this one.
>
> " java.io.IOException: Call to server1/10.85.85.17:9000 failed on local
> exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1107)
> at org.apache.hadoop.ipc.Client.call(Client.java:1075)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at $Proxy8.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
> at
> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
> at
>
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:176)
> at
>
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
> at
> org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:140)
> at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
> at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
> at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
> at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
> at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207)
> at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:205)
> at org.apache.spark.mllib.recommendation.ALS.run(ALS.scala:139)
> at org.apache.spark.mllib.recommendation.ALS$.main(ALS.scala:594)
> at org.apache.spark.mllib.recommendation.ALS.main(ALS.scala)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at
> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.ja
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/EOF-Exception-when-trying-to-access-hdfs-tp1347.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>