You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jeetendra Gangele <ga...@gmail.com> on 2015/04/12 22:16:19 UTC

regarding ZipWithIndex

Hi All I have an RDD JavaRDD<Object> and I want to convert it to
JavaPairRDD<Index,Object>.. Index should be unique and it should maintain
the order. For first object It should have 1 and then for second 2 like
that.

I tried using ZipWithIndex but it will return something like
JavaPairRDD<Object, long>
I wanted to use this RDD for lookup and join operation later in my workflow
so ordering is important.


Regards
jeet

Re: regarding ZipWithIndex

Posted by Jeetendra Gangele <ga...@gmail.com>.
How about using mapToPair and exchanging the two. Will it be efficient
Below is the code , will it be efficient to convert like this.


JavaPairRDD<Long, MatcherReleventData> RddForMarch
=matchRdd.zipWithindex.mapToPair(new
PairFunction<Tuple2<VendorRecord,Long>, Long, MatcherReleventData>() {

@Override
public Tuple2<Long, MatcherReleventData> call(Tuple2<VendorRecord, Long> t)
throws Exception {
MatcherReleventData matcherData = new MatcherReleventData();
Tuple2<Long, MatcherReleventData> tuple = new Tuple2<Long,
MatcherReleventData>(t._2,
matcherData.convertVendorDataToMatcherData(t._1));
 return tuple;
}

}).cache();

On 13 April 2015 at 03:11, Ted Yu <yu...@gmail.com> wrote:

> Please also take a look at ZippedWithIndexRDDPartition which is 72 lines
> long.
>
> You can create your own version which extends RDD[(Long, T)]
>
> Cheers
>
> On Sun, Apr 12, 2015 at 1:29 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> bq. will return something like JavaPairRDD<Object, long>
>>
>> The long component of the pair fits your description of index. What other
>> requirement does ZipWithIndex not provide you ?
>>
>> Cheers
>>
>> On Sun, Apr 12, 2015 at 1:16 PM, Jeetendra Gangele <ga...@gmail.com>
>> wrote:
>>
>>> Hi All I have an RDD JavaRDD<Object> and I want to convert it to
>>> JavaPairRDD<Index,Object>.. Index should be unique and it should maintain
>>> the order. For first object It should have 1 and then for second 2 like
>>> that.
>>>
>>> I tried using ZipWithIndex but it will return something like
>>> JavaPairRDD<Object, long>
>>> I wanted to use this RDD for lookup and join operation later in my
>>> workflow so ordering is important.
>>>
>>>
>>> Regards
>>> jeet
>>>
>>
>>
>

Re: regarding ZipWithIndex

Posted by Ted Yu <yu...@gmail.com>.
Please also take a look at ZippedWithIndexRDDPartition which is 72 lines
long.

You can create your own version which extends RDD[(Long, T)]

Cheers

On Sun, Apr 12, 2015 at 1:29 PM, Ted Yu <yu...@gmail.com> wrote:

> bq. will return something like JavaPairRDD<Object, long>
>
> The long component of the pair fits your description of index. What other
> requirement does ZipWithIndex not provide you ?
>
> Cheers
>
> On Sun, Apr 12, 2015 at 1:16 PM, Jeetendra Gangele <ga...@gmail.com>
> wrote:
>
>> Hi All I have an RDD JavaRDD<Object> and I want to convert it to
>> JavaPairRDD<Index,Object>.. Index should be unique and it should maintain
>> the order. For first object It should have 1 and then for second 2 like
>> that.
>>
>> I tried using ZipWithIndex but it will return something like
>> JavaPairRDD<Object, long>
>> I wanted to use this RDD for lookup and join operation later in my
>> workflow so ordering is important.
>>
>>
>> Regards
>> jeet
>>
>
>

Re: regarding ZipWithIndex

Posted by Jeetendra Gangele <ga...@gmail.com>.
Anyone who can guide me how to reduce the Size from Long to Int since I
dont need Long index.
I am huge data and this index talking 8 bytes, if i can reduce it to 4
bytes will be great help?

On 22 April 2015 at 22:46, Jeetendra Gangele <ga...@gmail.com> wrote:

> Sure thanks. if you can guide me how to do this will be great help.
>
> On 17 April 2015 at 22:05, Ted Yu <yu...@gmail.com> wrote:
>
>> I have some assignments on hand at the moment.
>>
>> Will try to come up with sample code after I clear the assignments.
>>
>> FYI
>>
>> On Thu, Apr 16, 2015 at 2:00 PM, Jeetendra Gangele <ga...@gmail.com>
>> wrote:
>>
>>> Can you please guide me how can I extend RDD and convert into this way
>>> you are suggesting.
>>>
>>> On 16 April 2015 at 23:46, Jeetendra Gangele <ga...@gmail.com>
>>> wrote:
>>>
>>>> I type T i already have Object ... I have RDD<Object> and then I am
>>>> calling ZipWithIndex on this RDD and getting RDD<Object,Long> on this I am
>>>> running MapToPair and converting into RDD<Long,Object> so that i can use it
>>>> later for other operation like lookup and join.
>>>>
>>>>
>>>> On 16 April 2015 at 23:42, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> The Long in RDD[(T, Long)] is type parameter. You can create RDD with
>>>>> Integer as the first type parameter.
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Thu, Apr 16, 2015 at 11:07 AM, Jeetendra Gangele <
>>>>> gangele397@gmail.com> wrote:
>>>>>
>>>>>> Hi Ted.
>>>>>> This works for me. But since Long takes here 8 bytes. Can I reduce it
>>>>>> to 4 bytes. its just a index and I feel 4 bytes was more than
>>>>>> enough.is there any method which takes Integer or similar for Index?
>>>>>>
>>>>>>
>>>>>> On 13 April 2015 at 01:59, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> bq. will return something like JavaPairRDD<Object, long>
>>>>>>>
>>>>>>> The long component of the pair fits your description of index. What
>>>>>>> other requirement does ZipWithIndex not provide you ?
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> On Sun, Apr 12, 2015 at 1:16 PM, Jeetendra Gangele <
>>>>>>> gangele397@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi All I have an RDD JavaRDD<Object> and I want to convert it to
>>>>>>>> JavaPairRDD<Index,Object>.. Index should be unique and it should maintain
>>>>>>>> the order. For first object It should have 1 and then for second 2 like
>>>>>>>> that.
>>>>>>>>
>>>>>>>> I tried using ZipWithIndex but it will return something like
>>>>>>>> JavaPairRDD<Object, long>
>>>>>>>> I wanted to use this RDD for lookup and join operation later in my
>>>>>>>> workflow so ordering is important.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> jeet
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>
>
>
>

Re: regarding ZipWithIndex

Posted by Jeetendra Gangele <ga...@gmail.com>.
Sure thanks. if you can guide me how to do this will be great help.

On 17 April 2015 at 22:05, Ted Yu <yu...@gmail.com> wrote:

> I have some assignments on hand at the moment.
>
> Will try to come up with sample code after I clear the assignments.
>
> FYI
>
> On Thu, Apr 16, 2015 at 2:00 PM, Jeetendra Gangele <ga...@gmail.com>
> wrote:
>
>> Can you please guide me how can I extend RDD and convert into this way
>> you are suggesting.
>>
>> On 16 April 2015 at 23:46, Jeetendra Gangele <ga...@gmail.com>
>> wrote:
>>
>>> I type T i already have Object ... I have RDD<Object> and then I am
>>> calling ZipWithIndex on this RDD and getting RDD<Object,Long> on this I am
>>> running MapToPair and converting into RDD<Long,Object> so that i can use it
>>> later for other operation like lookup and join.
>>>
>>>
>>> On 16 April 2015 at 23:42, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> The Long in RDD[(T, Long)] is type parameter. You can create RDD with
>>>> Integer as the first type parameter.
>>>>
>>>> Cheers
>>>>
>>>> On Thu, Apr 16, 2015 at 11:07 AM, Jeetendra Gangele <
>>>> gangele397@gmail.com> wrote:
>>>>
>>>>> Hi Ted.
>>>>> This works for me. But since Long takes here 8 bytes. Can I reduce it
>>>>> to 4 bytes. its just a index and I feel 4 bytes was more than
>>>>> enough.is there any method which takes Integer or similar for Index?
>>>>>
>>>>>
>>>>> On 13 April 2015 at 01:59, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> bq. will return something like JavaPairRDD<Object, long>
>>>>>>
>>>>>> The long component of the pair fits your description of index. What
>>>>>> other requirement does ZipWithIndex not provide you ?
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> On Sun, Apr 12, 2015 at 1:16 PM, Jeetendra Gangele <
>>>>>> gangele397@gmail.com> wrote:
>>>>>>
>>>>>>> Hi All I have an RDD JavaRDD<Object> and I want to convert it to
>>>>>>> JavaPairRDD<Index,Object>.. Index should be unique and it should maintain
>>>>>>> the order. For first object It should have 1 and then for second 2 like
>>>>>>> that.
>>>>>>>
>>>>>>> I tried using ZipWithIndex but it will return something like
>>>>>>> JavaPairRDD<Object, long>
>>>>>>> I wanted to use this RDD for lookup and join operation later in my
>>>>>>> workflow so ordering is important.
>>>>>>>
>>>>>>>
>>>>>>> Regards
>>>>>>> jeet
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>>
>

Re: regarding ZipWithIndex

Posted by Ted Yu <yu...@gmail.com>.
I have some assignments on hand at the moment.

Will try to come up with sample code after I clear the assignments.

FYI

On Thu, Apr 16, 2015 at 2:00 PM, Jeetendra Gangele <ga...@gmail.com>
wrote:

> Can you please guide me how can I extend RDD and convert into this way you
> are suggesting.
>
> On 16 April 2015 at 23:46, Jeetendra Gangele <ga...@gmail.com> wrote:
>
>> I type T i already have Object ... I have RDD<Object> and then I am
>> calling ZipWithIndex on this RDD and getting RDD<Object,Long> on this I am
>> running MapToPair and converting into RDD<Long,Object> so that i can use it
>> later for other operation like lookup and join.
>>
>>
>> On 16 April 2015 at 23:42, Ted Yu <yu...@gmail.com> wrote:
>>
>>> The Long in RDD[(T, Long)] is type parameter. You can create RDD with
>>> Integer as the first type parameter.
>>>
>>> Cheers
>>>
>>> On Thu, Apr 16, 2015 at 11:07 AM, Jeetendra Gangele <
>>> gangele397@gmail.com> wrote:
>>>
>>>> Hi Ted.
>>>> This works for me. But since Long takes here 8 bytes. Can I reduce it
>>>> to 4 bytes. its just a index and I feel 4 bytes was more than enough.is
>>>> there any method which takes Integer or similar for Index?
>>>>
>>>>
>>>> On 13 April 2015 at 01:59, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> bq. will return something like JavaPairRDD<Object, long>
>>>>>
>>>>> The long component of the pair fits your description of index. What
>>>>> other requirement does ZipWithIndex not provide you ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Sun, Apr 12, 2015 at 1:16 PM, Jeetendra Gangele <
>>>>> gangele397@gmail.com> wrote:
>>>>>
>>>>>> Hi All I have an RDD JavaRDD<Object> and I want to convert it to
>>>>>> JavaPairRDD<Index,Object>.. Index should be unique and it should maintain
>>>>>> the order. For first object It should have 1 and then for second 2 like
>>>>>> that.
>>>>>>
>>>>>> I tried using ZipWithIndex but it will return something like
>>>>>> JavaPairRDD<Object, long>
>>>>>> I wanted to use this RDD for lookup and join operation later in my
>>>>>> workflow so ordering is important.
>>>>>>
>>>>>>
>>>>>> Regards
>>>>>> jeet
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
>
>
>

Re: regarding ZipWithIndex

Posted by Jeetendra Gangele <ga...@gmail.com>.
Hi Ted

Will you be able to guide me for any sample code for achieving this?

On 17 April 2015 at 02:30, Jeetendra Gangele <ga...@gmail.com> wrote:

> Can you please guide me how can I extend RDD and convert into this way you
> are suggesting.
>
> On 16 April 2015 at 23:46, Jeetendra Gangele <ga...@gmail.com> wrote:
>
>> I type T i already have Object ... I have RDD<Object> and then I am
>> calling ZipWithIndex on this RDD and getting RDD<Object,Long> on this I am
>> running MapToPair and converting into RDD<Long,Object> so that i can use it
>> later for other operation like lookup and join.
>>
>>
>> On 16 April 2015 at 23:42, Ted Yu <yu...@gmail.com> wrote:
>>
>>> The Long in RDD[(T, Long)] is type parameter. You can create RDD with
>>> Integer as the first type parameter.
>>>
>>> Cheers
>>>
>>> On Thu, Apr 16, 2015 at 11:07 AM, Jeetendra Gangele <
>>> gangele397@gmail.com> wrote:
>>>
>>>> Hi Ted.
>>>> This works for me. But since Long takes here 8 bytes. Can I reduce it
>>>> to 4 bytes. its just a index and I feel 4 bytes was more than enough.is
>>>> there any method which takes Integer or similar for Index?
>>>>
>>>>
>>>> On 13 April 2015 at 01:59, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> bq. will return something like JavaPairRDD<Object, long>
>>>>>
>>>>> The long component of the pair fits your description of index. What
>>>>> other requirement does ZipWithIndex not provide you ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Sun, Apr 12, 2015 at 1:16 PM, Jeetendra Gangele <
>>>>> gangele397@gmail.com> wrote:
>>>>>
>>>>>> Hi All I have an RDD JavaRDD<Object> and I want to convert it to
>>>>>> JavaPairRDD<Index,Object>.. Index should be unique and it should maintain
>>>>>> the order. For first object It should have 1 and then for second 2 like
>>>>>> that.
>>>>>>
>>>>>> I tried using ZipWithIndex but it will return something like
>>>>>> JavaPairRDD<Object, long>
>>>>>> I wanted to use this RDD for lookup and join operation later in my
>>>>>> workflow so ordering is important.
>>>>>>
>>>>>>
>>>>>> Regards
>>>>>> jeet
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
>
>
>

Re: regarding ZipWithIndex

Posted by Jeetendra Gangele <ga...@gmail.com>.
Can you please guide me how can I extend RDD and convert into this way you
are suggesting.

On 16 April 2015 at 23:46, Jeetendra Gangele <ga...@gmail.com> wrote:

> I type T i already have Object ... I have RDD<Object> and then I am
> calling ZipWithIndex on this RDD and getting RDD<Object,Long> on this I am
> running MapToPair and converting into RDD<Long,Object> so that i can use it
> later for other operation like lookup and join.
>
>
> On 16 April 2015 at 23:42, Ted Yu <yu...@gmail.com> wrote:
>
>> The Long in RDD[(T, Long)] is type parameter. You can create RDD with
>> Integer as the first type parameter.
>>
>> Cheers
>>
>> On Thu, Apr 16, 2015 at 11:07 AM, Jeetendra Gangele <gangele397@gmail.com
>> > wrote:
>>
>>> Hi Ted.
>>> This works for me. But since Long takes here 8 bytes. Can I reduce it to
>>> 4 bytes. its just a index and I feel 4 bytes was more than enough.is
>>> there any method which takes Integer or similar for Index?
>>>
>>>
>>> On 13 April 2015 at 01:59, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> bq. will return something like JavaPairRDD<Object, long>
>>>>
>>>> The long component of the pair fits your description of index. What
>>>> other requirement does ZipWithIndex not provide you ?
>>>>
>>>> Cheers
>>>>
>>>> On Sun, Apr 12, 2015 at 1:16 PM, Jeetendra Gangele <
>>>> gangele397@gmail.com> wrote:
>>>>
>>>>> Hi All I have an RDD JavaRDD<Object> and I want to convert it to
>>>>> JavaPairRDD<Index,Object>.. Index should be unique and it should maintain
>>>>> the order. For first object It should have 1 and then for second 2 like
>>>>> that.
>>>>>
>>>>> I tried using ZipWithIndex but it will return something like
>>>>> JavaPairRDD<Object, long>
>>>>> I wanted to use this RDD for lookup and join operation later in my
>>>>> workflow so ordering is important.
>>>>>
>>>>>
>>>>> Regards
>>>>> jeet
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>
>
>

Re: regarding ZipWithIndex

Posted by Jeetendra Gangele <ga...@gmail.com>.
I type T i already have Object ... I have RDD<Object> and then I am calling
ZipWithIndex on this RDD and getting RDD<Object,Long> on this I am running
MapToPair and converting into RDD<Long,Object> so that i can use it later
for other operation like lookup and join.

On 16 April 2015 at 23:42, Ted Yu <yu...@gmail.com> wrote:

> The Long in RDD[(T, Long)] is type parameter. You can create RDD with
> Integer as the first type parameter.
>
> Cheers
>
> On Thu, Apr 16, 2015 at 11:07 AM, Jeetendra Gangele <ga...@gmail.com>
> wrote:
>
>> Hi Ted.
>> This works for me. But since Long takes here 8 bytes. Can I reduce it to
>> 4 bytes. its just a index and I feel 4 bytes was more than enough.is
>> there any method which takes Integer or similar for Index?
>>
>>
>> On 13 April 2015 at 01:59, Ted Yu <yu...@gmail.com> wrote:
>>
>>> bq. will return something like JavaPairRDD<Object, long>
>>>
>>> The long component of the pair fits your description of index. What
>>> other requirement does ZipWithIndex not provide you ?
>>>
>>> Cheers
>>>
>>> On Sun, Apr 12, 2015 at 1:16 PM, Jeetendra Gangele <gangele397@gmail.com
>>> > wrote:
>>>
>>>> Hi All I have an RDD JavaRDD<Object> and I want to convert it to
>>>> JavaPairRDD<Index,Object>.. Index should be unique and it should maintain
>>>> the order. For first object It should have 1 and then for second 2 like
>>>> that.
>>>>
>>>> I tried using ZipWithIndex but it will return something like
>>>> JavaPairRDD<Object, long>
>>>> I wanted to use this RDD for lookup and join operation later in my
>>>> workflow so ordering is important.
>>>>
>>>>
>>>> Regards
>>>> jeet
>>>>
>>>
>>>
>>
>>
>>
>

Re: regarding ZipWithIndex

Posted by Ted Yu <yu...@gmail.com>.
The Long in RDD[(T, Long)] is type parameter. You can create RDD with
Integer as the first type parameter.

Cheers

On Thu, Apr 16, 2015 at 11:07 AM, Jeetendra Gangele <ga...@gmail.com>
wrote:

> Hi Ted.
> This works for me. But since Long takes here 8 bytes. Can I reduce it to 4
> bytes. its just a index and I feel 4 bytes was more than enough.is there
> any method which takes Integer or similar for Index?
>
>
> On 13 April 2015 at 01:59, Ted Yu <yu...@gmail.com> wrote:
>
>> bq. will return something like JavaPairRDD<Object, long>
>>
>> The long component of the pair fits your description of index. What other
>> requirement does ZipWithIndex not provide you ?
>>
>> Cheers
>>
>> On Sun, Apr 12, 2015 at 1:16 PM, Jeetendra Gangele <ga...@gmail.com>
>> wrote:
>>
>>> Hi All I have an RDD JavaRDD<Object> and I want to convert it to
>>> JavaPairRDD<Index,Object>.. Index should be unique and it should maintain
>>> the order. For first object It should have 1 and then for second 2 like
>>> that.
>>>
>>> I tried using ZipWithIndex but it will return something like
>>> JavaPairRDD<Object, long>
>>> I wanted to use this RDD for lookup and join operation later in my
>>> workflow so ordering is important.
>>>
>>>
>>> Regards
>>> jeet
>>>
>>
>>
>
>
>

Re: regarding ZipWithIndex

Posted by Jeetendra Gangele <ga...@gmail.com>.
Hi Ted.
This works for me. But since Long takes here 8 bytes. Can I reduce it to 4
bytes. its just a index and I feel 4 bytes was more than enough.is there
any method which takes Integer or similar for Index?


On 13 April 2015 at 01:59, Ted Yu <yu...@gmail.com> wrote:

> bq. will return something like JavaPairRDD<Object, long>
>
> The long component of the pair fits your description of index. What other
> requirement does ZipWithIndex not provide you ?
>
> Cheers
>
> On Sun, Apr 12, 2015 at 1:16 PM, Jeetendra Gangele <ga...@gmail.com>
> wrote:
>
>> Hi All I have an RDD JavaRDD<Object> and I want to convert it to
>> JavaPairRDD<Index,Object>.. Index should be unique and it should maintain
>> the order. For first object It should have 1 and then for second 2 like
>> that.
>>
>> I tried using ZipWithIndex but it will return something like
>> JavaPairRDD<Object, long>
>> I wanted to use this RDD for lookup and join operation later in my
>> workflow so ordering is important.
>>
>>
>> Regards
>> jeet
>>
>
>

Re: regarding ZipWithIndex

Posted by Ted Yu <yu...@gmail.com>.
bq. will return something like JavaPairRDD<Object, long>

The long component of the pair fits your description of index. What other
requirement does ZipWithIndex not provide you ?

Cheers

On Sun, Apr 12, 2015 at 1:16 PM, Jeetendra Gangele <ga...@gmail.com>
wrote:

> Hi All I have an RDD JavaRDD<Object> and I want to convert it to
> JavaPairRDD<Index,Object>.. Index should be unique and it should maintain
> the order. For first object It should have 1 and then for second 2 like
> that.
>
> I tried using ZipWithIndex but it will return something like
> JavaPairRDD<Object, long>
> I wanted to use this RDD for lookup and join operation later in my
> workflow so ordering is important.
>
>
> Regards
> jeet
>