You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jaonary Rabarisoa <ja...@gmail.com> on 2014/10/17 08:58:47 UTC

key class requirement for PairedRDD ?

Dear all,

Is it possible to use any kind of object as key in a PairedRDD. When I use
a case class  key, the groupByKey operation don't behave as I expected. I
want to use a case class to avoid using a large tuple as it is easier to
manipulate.


Cheers,

Jaonary

Re: key class requirement for PairedRDD ?

Posted by Jaonary Rabarisoa <ja...@gmail.com>.
Here what I'm trying to do. My case class is the following :

case class PersonID(id: String, group: String, name: String)

I want to use PersonID as a key in a PairedRDD. But I think the default
equal function don't fit to my need because two PersonID("a","a","a") are
not the same. When I use a tuple (String, String, String) as a key it's
working.

On Fri, Oct 17, 2014 at 9:03 AM, Sonal Goyal <so...@gmail.com> wrote:

> We use our custom classes which are Serializable and have well defined
> hashcode and equals methods through the Java API. Whats the issue you are
> getting?
>
> Best Regards,
> Sonal
> Nube Technologies <http://www.nubetech.co>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
> On Fri, Oct 17, 2014 at 12:28 PM, Jaonary Rabarisoa <ja...@gmail.com>
> wrote:
>
>> Dear all,
>>
>> Is it possible to use any kind of object as key in a PairedRDD. When I
>> use a case class  key, the groupByKey operation don't behave as I expected.
>> I want to use a case class to avoid using a large tuple as it is easier to
>> manipulate.
>>
>>
>> Cheers,
>>
>> Jaonary
>>
>
>

Re: key class requirement for PairedRDD ?

Posted by Sonal Goyal <so...@gmail.com>.
We use our custom classes which are Serializable and have well defined
hashcode and equals methods through the Java API. Whats the issue you are
getting?

Best Regards,
Sonal
Nube Technologies <http://www.nubetech.co>

<http://in.linkedin.com/in/sonalgoyal>



On Fri, Oct 17, 2014 at 12:28 PM, Jaonary Rabarisoa <ja...@gmail.com>
wrote:

> Dear all,
>
> Is it possible to use any kind of object as key in a PairedRDD. When I use
> a case class  key, the groupByKey operation don't behave as I expected. I
> want to use a case class to avoid using a large tuple as it is easier to
> manipulate.
>
>
> Cheers,
>
> Jaonary
>