You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Archit Thakur <ar...@gmail.com> on 2013/12/09 05:58:45 UTC
Key Class - NotSerializableException
Hi,
When I did
sc.sequenceFile(file, classOf[Text],
classOf[Text]).flatMap(map_func).count()
It gave me result of 365.
However, when I did
sc.sequenceFile(file, classOf[Text],
classOf[Text]).flatMap(map_func).sortByKey().count(),
It threw java.io.NotSerializableException for Key Class returned by
flapMap. My question is
Why does sortByKey require the Key/Value Classes to be Serialized.?
Thanks and Regards,
Archit Thakur.
Re: Key Class - NotSerializableException
Posted by Archit Thakur <ar...@gmail.com>.
1 update:
Now both of the commands with and without sortByKey have started throwing
an error.
Loss was due to java.lang.NoSuchMethodError
java.lang.NoSuchMethodError:
com.guavus.logging.Logger.<init>(Ljava/lang/Class;)V
at
com.guavus.mapred.common.collection.Collection.<clinit>(Collection.java:17)
On Mon, Dec 9, 2013 at 11:21 AM, Archit Thakur <ar...@gmail.com>wrote:
> And Since sortByKey serializes the classes, I guess it has something to do
> with Serialization thing.
>
>
> On Mon, Dec 9, 2013 at 11:19 AM, Archit Thakur <ar...@gmail.com>wrote:
>
>> I did make the classes Serialized. But now running the same command
>> sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_
>> func).sortByKey().count(), gives me java.lang.NoSuchMethodError.
>>
>> For the Collection class which I made Serialized accesses one static
>> variable that
>>
>> static com.xyz.logging.Logger Logger = new
>> com.xyz.logging.Logger(Collection.class) and It throws
>>
>> java.lang.NoSuchMethodError:
>> com.guavus.logging.Logger.<init>(Ljava/lang/Class;)V
>> at
>> com.guavus.mapred.common.collection.Collection.<clinit>(Collection.java:17)
>>
>> but it doesn't do that when I don't sortByKey, ie when I run
>> sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_
>> func).count() it doesn't throw the error.
>>
>> Thanks and Regards,
>> Archit Thakur.
>>
>>
>>
>> On Mon, Dec 9, 2013 at 10:48 AM, Patrick Wendell <pw...@gmail.com>wrote:
>>
>>> It's because sorting serializes the data during the shuffle phase.
>>>
>>> On Sun, Dec 8, 2013 at 8:58 PM, Archit Thakur <ar...@gmail.com>
>>> wrote:
>>> > Hi,
>>> >
>>> > When I did
>>> >
>>> > sc.sequenceFile(file, classOf[Text],
>>> > classOf[Text]).flatMap(map_func).count()
>>> > It gave me result of 365.
>>> >
>>> > However, when I did
>>> > sc.sequenceFile(file, classOf[Text],
>>> > classOf[Text]).flatMap(map_func).sortByKey().count(),
>>> >
>>> > It threw java.io.NotSerializableException for Key Class returned by
>>> flapMap.
>>> > My question is
>>> > Why does sortByKey require the Key/Value Classes to be Serialized.?
>>> >
>>> > Thanks and Regards,
>>> > Archit Thakur.
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> Groups
>>> > "Spark Users" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> an
>>> > email to spark-users+unsubscribe@googlegroups.com.
>>> > For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Spark Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to spark-users+unsubscribe@googlegroups.com.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>
>
Fwd: Key Class - NotSerializableException
Posted by Archit Thakur <ar...@gmail.com>.
Hi Nick,
Yeah I saw that. I actually used sc.sequenceFile file to load data from
HDFS to RDD. Also both my key class and value class implements
WritableComparable of Hadoop. Still I got the error
"java.io.NotSerializableException", When I used sortByKey.
Hierarchy of my classes:
Collection
KeyCollection extends Collection implements WritableComparable
ValueCollection extends Collection implements WritableComparable
DS extends KeyCollection
MS extends ValueCollection
and I use DS and MS classes for key and value. With this hierarchy I get
java.io.NotSerializableException with sortByKey. So I made Collection as
Serializable and now It was unable to find some method required for the
static field of class Collection.
Thanks and Regards,
Archit Thakur.
On Mon, Dec 9, 2013 at 11:38 AM, MLnick <ni...@gmail.com> wrote:
> Hi Archit
>
> Spark provides a convenience class for sequencefile that provides implicit
> conversion from Writable to appropriate Scala classes:
>
> Import SparkContext._
> sc.sequenceFile[String, String](path)
>
> You should end up with an RDD[(String, String)] and won't have any
> serializable issues.
>
> Hope this helps
> N
>
> --
> You received this message because you are subscribed to the Google Groups
> "Spark Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spark-users+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
Re: Key Class - NotSerializableException
Posted by Archit Thakur <ar...@gmail.com>.
And Since sortByKey serializes the classes, I guess it has something to do
with Serialization thing.
On Mon, Dec 9, 2013 at 11:19 AM, Archit Thakur <ar...@gmail.com>wrote:
> I did make the classes Serialized. But now running the same command
> sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_
> func).sortByKey().count(), gives me java.lang.NoSuchMethodError.
>
> For the Collection class which I made Serialized accesses one static
> variable that
>
> static com.xyz.logging.Logger Logger = new
> com.xyz.logging.Logger(Collection.class) and It throws
>
> java.lang.NoSuchMethodError:
> com.guavus.logging.Logger.<init>(Ljava/lang/Class;)V
> at
> com.guavus.mapred.common.collection.Collection.<clinit>(Collection.java:17)
>
> but it doesn't do that when I don't sortByKey, ie when I run
> sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_
> func).count() it doesn't throw the error.
>
> Thanks and Regards,
> Archit Thakur.
>
>
>
> On Mon, Dec 9, 2013 at 10:48 AM, Patrick Wendell <pw...@gmail.com>wrote:
>
>> It's because sorting serializes the data during the shuffle phase.
>>
>> On Sun, Dec 8, 2013 at 8:58 PM, Archit Thakur <ar...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > When I did
>> >
>> > sc.sequenceFile(file, classOf[Text],
>> > classOf[Text]).flatMap(map_func).count()
>> > It gave me result of 365.
>> >
>> > However, when I did
>> > sc.sequenceFile(file, classOf[Text],
>> > classOf[Text]).flatMap(map_func).sortByKey().count(),
>> >
>> > It threw java.io.NotSerializableException for Key Class returned by
>> flapMap.
>> > My question is
>> > Why does sortByKey require the Key/Value Classes to be Serialized.?
>> >
>> > Thanks and Regards,
>> > Archit Thakur.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups
>> > "Spark Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an
>> > email to spark-users+unsubscribe@googlegroups.com.
>> > For more options, visit https://groups.google.com/groups/opt_out.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Spark Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to spark-users+unsubscribe@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
Re: Key Class - NotSerializableException
Posted by Archit Thakur <ar...@gmail.com>.
I did make the classes Serialized. But now running the same command
sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_
func).sortByKey().count(), gives me java.lang.NoSuchMethodError.
For the Collection class which I made Serialized accesses one static
variable that
static com.xyz.logging.Logger Logger = new
com.xyz.logging.Logger(Collection.class) and It throws
java.lang.NoSuchMethodError:
com.guavus.logging.Logger.<init>(Ljava/lang/Class;)V
at
com.guavus.mapred.common.collection.Collection.<clinit>(Collection.java:17)
but it doesn't do that when I don't sortByKey, ie when I run
sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_
func).count() it doesn't throw the error.
Thanks and Regards,
Archit Thakur.
On Mon, Dec 9, 2013 at 10:48 AM, Patrick Wendell <pw...@gmail.com> wrote:
> It's because sorting serializes the data during the shuffle phase.
>
> On Sun, Dec 8, 2013 at 8:58 PM, Archit Thakur <ar...@gmail.com>
> wrote:
> > Hi,
> >
> > When I did
> >
> > sc.sequenceFile(file, classOf[Text],
> > classOf[Text]).flatMap(map_func).count()
> > It gave me result of 365.
> >
> > However, when I did
> > sc.sequenceFile(file, classOf[Text],
> > classOf[Text]).flatMap(map_func).sortByKey().count(),
> >
> > It threw java.io.NotSerializableException for Key Class returned by
> flapMap.
> > My question is
> > Why does sortByKey require the Key/Value Classes to be Serialized.?
> >
> > Thanks and Regards,
> > Archit Thakur.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Spark Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to spark-users+unsubscribe@googlegroups.com.
> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Spark Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spark-users+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
Re: Key Class - NotSerializableException
Posted by Patrick Wendell <pw...@gmail.com>.
It's because sorting serializes the data during the shuffle phase.
On Sun, Dec 8, 2013 at 8:58 PM, Archit Thakur <ar...@gmail.com> wrote:
> Hi,
>
> When I did
>
> sc.sequenceFile(file, classOf[Text],
> classOf[Text]).flatMap(map_func).count()
> It gave me result of 365.
>
> However, when I did
> sc.sequenceFile(file, classOf[Text],
> classOf[Text]).flatMap(map_func).sortByKey().count(),
>
> It threw java.io.NotSerializableException for Key Class returned by flapMap.
> My question is
> Why does sortByKey require the Key/Value Classes to be Serialized.?
>
> Thanks and Regards,
> Archit Thakur.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Spark Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spark-users+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.