You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Alexander Pivovarov <ap...@gmail.com> on 2016/06/08 23:53:28 UTC
Kryo registration for Tuples?
if my RDD is RDD[(String, (Long, MyClass))]
Do I need to register
classOf[MyClass]
classOf[(Any, Any)]
or
classOf[MyClass]
classOf[(Long, MyClass)]
classOf[(String, (Long, MyClass))]
?
Re: Kryo registration for Tuples?
Posted by Reynold Xin <rx...@databricks.com>.
Yes you can :)
On Wed, Jun 8, 2016 at 6:00 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:
> Can I just enable spark.kryo.registrationRequired and look at error
> messages to get unregistered classes?
>
> On Wed, Jun 8, 2016 at 5:52 PM, Reynold Xin <rx...@databricks.com> wrote:
>
>> Due to type erasure they have no difference, although watch out for Scala
>> tuple serialization.
>>
>>
>> On Wednesday, June 8, 2016, Ted Yu <yu...@gmail.com> wrote:
>>
>>> I think the second group (3 classOf's) should be used.
>>>
>>> Cheers
>>>
>>> On Wed, Jun 8, 2016 at 4:53 PM, Alexander Pivovarov <
>>> apivovarov@gmail.com> wrote:
>>>
>>>> if my RDD is RDD[(String, (Long, MyClass))]
>>>>
>>>> Do I need to register
>>>>
>>>> classOf[MyClass]
>>>> classOf[(Any, Any)]
>>>>
>>>> or
>>>>
>>>> classOf[MyClass]
>>>> classOf[(Long, MyClass)]
>>>> classOf[(String, (Long, MyClass))]
>>>>
>>>> ?
>>>>
>>>>
>>>
>
Re: Kryo registration for Tuples?
Posted by Alexander Pivovarov <ap...@gmail.com>.
Can I just enable spark.kryo.registrationRequired and look at error
messages to get unregistered classes?
On Wed, Jun 8, 2016 at 5:52 PM, Reynold Xin <rx...@databricks.com> wrote:
> Due to type erasure they have no difference, although watch out for Scala
> tuple serialization.
>
>
> On Wednesday, June 8, 2016, Ted Yu <yu...@gmail.com> wrote:
>
>> I think the second group (3 classOf's) should be used.
>>
>> Cheers
>>
>> On Wed, Jun 8, 2016 at 4:53 PM, Alexander Pivovarov <apivovarov@gmail.com
>> > wrote:
>>
>>> if my RDD is RDD[(String, (Long, MyClass))]
>>>
>>> Do I need to register
>>>
>>> classOf[MyClass]
>>> classOf[(Any, Any)]
>>>
>>> or
>>>
>>> classOf[MyClass]
>>> classOf[(Long, MyClass)]
>>> classOf[(String, (Long, MyClass))]
>>>
>>> ?
>>>
>>>
>>
Re: Kryo registration for Tuples?
Posted by Reynold Xin <rx...@databricks.com>.
Due to type erasure they have no difference, although watch out for Scala
tuple serialization.
On Wednesday, June 8, 2016, Ted Yu <yu...@gmail.com> wrote:
> I think the second group (3 classOf's) should be used.
>
> Cheers
>
> On Wed, Jun 8, 2016 at 4:53 PM, Alexander Pivovarov <apivovarov@gmail.com
> <javascript:_e(%7B%7D,'cvml','apivovarov@gmail.com');>> wrote:
>
>> if my RDD is RDD[(String, (Long, MyClass))]
>>
>> Do I need to register
>>
>> classOf[MyClass]
>> classOf[(Any, Any)]
>>
>> or
>>
>> classOf[MyClass]
>> classOf[(Long, MyClass)]
>> classOf[(String, (Long, MyClass))]
>>
>> ?
>>
>>
>
Re: Kryo registration for Tuples?
Posted by Ted Yu <yu...@gmail.com>.
I think the second group (3 classOf's) should be used.
Cheers
On Wed, Jun 8, 2016 at 4:53 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:
> if my RDD is RDD[(String, (Long, MyClass))]
>
> Do I need to register
>
> classOf[MyClass]
> classOf[(Any, Any)]
>
> or
>
> classOf[MyClass]
> classOf[(Long, MyClass)]
> classOf[(String, (Long, MyClass))]
>
> ?
>
>