ClassCastException when using saveAsTextFile

[Niko Stahl <r....@gmail.com>]

Hi,

I'm trying to save an RDD to HDFS with the saveAsTextFile method on my ec2
cluster and am encountering the following exception (the app is called
GraphTest):

Exception failure: java.lang.ClassCastException: cannot assign instance of
GraphTest$$anonfun$3 to field org.apache.spark.rdd.MappedRDD.f of type
scala.Function1 in instance of org.apache.spark.rdd.MappedRDD

The RDD is simply a list of strings. Strangely enough the same sequence of
commands when executed in the Spark shell does not cause the above error.
Any thoughts on what might be going on here?

Thanks,
Niko

Re: ClassCastException when using saveAsTextFile

[Anwar Rizal <an...@gmail.com>]

Hi Niko,

I execute the script in 0.9/CDH5 using spark-shell , and it does not
generate ClassCastException. Which version are you using and can you give
more  stack trace ?

Cheers,
a.


On Tue, Mar 25, 2014 at 7:55 PM, Niko Stahl <r....@gmail.com> wrote:

> Ok, so I've been able to narrow down the problem to this specific case:
>
> def toCsv(userTuple: String) = {"a,b,c"}
> val dataTemp = Array("line1", "line2")
> val dataTempDist = sc.parallelize(dataTemp)
> val usersFormatted = dataTempDist.map(toCsv)
> usersFormatted.saveAsTextFile("hdfs://" + masterDomain +
> ":9000/user/root/" +  "test_dir")
>
> Even this simple mapping give me a java.lang.ClassCastException. Sorry,
> my knowledge of Scala is very rudimentary.
>
> Thanks,
> Niko
>
>
> On Tue, Mar 25, 2014 at 5:55 PM, Niko Stahl <r....@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm trying to save an RDD to HDFS with the saveAsTextFile method on my
>> ec2 cluster and am encountering the following exception (the app is called
>> GraphTest):
>>
>> Exception failure: java.lang.ClassCastException: cannot assign instance
>> of GraphTest$$anonfun$3 to field org.apache.spark.rdd.MappedRDD.f of type
>> scala.Function1 in instance of org.apache.spark.rdd.MappedRDD
>>
>> The RDD is simply a list of strings. Strangely enough the same sequence
>> of commands when executed in the Spark shell does not cause the above
>> error. Any thoughts on what might be going on here?
>>
>> Thanks,
>> Niko
>>
>
>

Re: ClassCastException when using saveAsTextFile

[Kanwaldeep <ka...@gmail.com>]

Hi Niko

I'm having a similar problem with running the Sparks on standalone cluster.
Any suggestions on how to fix this? The error is happening on using
pairRDDFunction saveAsHadoopDataSet.

java.lang.ClassCastException (java.lang.ClassCastException: cannot assign
instance of org.apache.spark.rdd.PairRDDFunctions$$anonfun$combineByKey$1 to
field org.apache.spark.rdd.RDD$$anonfun$3.f$7 of type scala.Function2 in
instance of org.apache.spark.rdd.RDD$$anonfun$3)

Thanks
Kanwal



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ClassCastException-when-using-saveAsTextFile-tp3206p7018.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: ClassCastException when using saveAsTextFile

[Niko Stahl <r....@gmail.com>]

Ok, so I've been able to narrow down the problem to this specific case:

def toCsv(userTuple: String) = {"a,b,c"}
val dataTemp = Array("line1", "line2")
val dataTempDist = sc.parallelize(dataTemp)
val usersFormatted = dataTempDist.map(toCsv)
usersFormatted.saveAsTextFile("hdfs://" + masterDomain + ":9000/user/root/"
+  "test_dir")

Even this simple mapping give me a java.lang.ClassCastException. Sorry, my
knowledge of Scala is very rudimentary.

Thanks,
Niko


On Tue, Mar 25, 2014 at 5:55 PM, Niko Stahl <r....@gmail.com> wrote:

> Hi,
>
> I'm trying to save an RDD to HDFS with the saveAsTextFile method on my ec2
> cluster and am encountering the following exception (the app is called
> GraphTest):
>
> Exception failure: java.lang.ClassCastException: cannot assign instance of
> GraphTest$$anonfun$3 to field org.apache.spark.rdd.MappedRDD.f of type
> scala.Function1 in instance of org.apache.spark.rdd.MappedRDD
>
> The RDD is simply a list of strings. Strangely enough the same sequence of
> commands when executed in the Spark shell does not cause the above error.
> Any thoughts on what might be going on here?
>
> Thanks,
> Niko
>