You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by kmader <ke...@gmail.com> on 2014/07/23 17:25:32 UTC

Re: wholeTextFiles not working with HDFS

I have the same issue

        val a = sc.textFile("s3n://MyBucket/MyFolder/*.tif")
        a.first

works perfectly fine, but 

        val d = sc.wholeTextFiles("s3n://MyBucket/MyFolder/*.tif")  does not
work
        d.first

Gives the following error message

        java.io.FileNotFoundException: File /MyBucket/MyFolder.tif does not
exist.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/wholeTextFiles-not-working-with-HDFS-tp7490p10505.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: wholeTextFiles not working with HDFS

Posted by pierred <pi...@demartines.com>.
I forgot to say, I am using bin/spark-shell, spark-1.0.2
That host has scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
1.8.0_11)




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/wholeTextFiles-not-working-with-HDFS-tp7490p12678.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: wholeTextFiles not working with HDFS

Posted by pierred <pi...@demartines.com>.
I had the same issue with spark-1.0.2-bin-hadoop*1*, and indeed the issue
seems related to Hadoop1.  When switching to using
spark-1.0.2-bin-hadoop*2*, the issue disappears.




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/wholeTextFiles-not-working-with-HDFS-tp7490p12677.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: wholeTextFiles not working with HDFS

Posted by kmader <ke...@gmail.com>.
That worked for me as well, I was using spark 1.0 compiled against Hadoop
1.0, switching to 1.0.1 compiled against hadoop 2



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/wholeTextFiles-not-working-with-HDFS-tp7490p10547.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.