You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Amit Sela <am...@gmail.com> on 2016/05/22 21:50:31 UTC

Dataset kryo encoder fails on Collections$UnmodifiableCollection

I've been using Encoders with Kryo to support encoding of generically typed
Java classes, mostly with success, in the following manner:

public static <T> Encoder<T> encoder() {
  return Encoders.kryo((Class<T>) Object.class);
}

But at some point I got a decoding exception "Caused by:
java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableCollection.add..."

This seems to be because of Guava's `ImmutableList`.

I tried registering `UnmodifiableCollectionsSerializer` and `
ImmutableListSerializer` from: https://github.com/magro/kryo-serializers
but it didn't help.

Ideas ?

Thanks,
Amit

Re: Dataset kryo encoder fails on Collections$UnmodifiableCollection

Posted by Amit Sela <am...@gmail.com>.

See SPARK-15489 <https://issues.apache.org/jira/browse/SPARK-15489>
I'll try to figure this one out as well, any leads ? "immediate suspects" ?

Thanks,
Amit

On Mon, May 23, 2016 at 10:27 PM Michael Armbrust <mi...@databricks.com>
wrote:

> Can you open a JIRA?
>
> On Sun, May 22, 2016 at 2:50 PM, Amit Sela <am...@gmail.com> wrote:
>
>> I've been using Encoders with Kryo to support encoding of generically
>> typed Java classes, mostly with success, in the following manner:
>>
>> public static <T> Encoder<T> encoder() {
>>   return Encoders.kryo((Class<T>) Object.class);
>> }
>>
>> But at some point I got a decoding exception "Caused by:
>> java.lang.UnsupportedOperationException
>> at java.util.Collections$UnmodifiableCollection.add..."
>>
>> This seems to be because of Guava's `ImmutableList`.
>>
>> I tried registering `UnmodifiableCollectionsSerializer` and `
>> ImmutableListSerializer` from: https://github.com/magro/kryo-serializers
>> but it didn't help.
>>
>> Ideas ?
>>
>> Thanks,
>> Amit
>>
>
>

Re: Dataset kryo encoder fails on Collections$UnmodifiableCollection

Posted by Michael Armbrust <mi...@databricks.com>.

Can you open a JIRA?

On Sun, May 22, 2016 at 2:50 PM, Amit Sela <am...@gmail.com> wrote:

> I've been using Encoders with Kryo to support encoding of generically
> typed Java classes, mostly with success, in the following manner:
>
> public static <T> Encoder<T> encoder() {
>   return Encoders.kryo((Class<T>) Object.class);
> }
>
> But at some point I got a decoding exception "Caused by:
> java.lang.UnsupportedOperationException
> at java.util.Collections$UnmodifiableCollection.add..."
>
> This seems to be because of Guava's `ImmutableList`.
>
> I tried registering `UnmodifiableCollectionsSerializer` and `
> ImmutableListSerializer` from: https://github.com/magro/kryo-serializers
> but it didn't help.
>
> Ideas ?
>
> Thanks,
> Amit
>