You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Soumitra Kumar <ku...@gmail.com> on 2014/08/27 23:37:27 UTC

Spark Streaming: DStream - zipWithIndex

Hello,

If I do:

DStream transform {
    rdd.zipWithIndex.map {

        Is the index guaranteed to be unique across all RDDs here?

}
}

Thanks,
-Soumitra.

Re: Spark Streaming: DStream - zipWithIndex

Posted by Patrick Wendell <pw...@gmail.com>.

Yeah - each batch will produce a new RDD.

On Wed, Aug 27, 2014 at 3:33 PM, Soumitra Kumar
<ku...@gmail.com> wrote:
> Thanks.
>
> Just to double check, rdd.id would be unique for a batch in a DStream?
>
>
> On Wed, Aug 27, 2014 at 3:04 PM, Xiangrui Meng <me...@gmail.com> wrote:
>>
>> You can use RDD id as the seed, which is unique in the same spark
>> context. Suppose none of the RDDs would contain more than 1 billion
>> records. Then you can use
>>
>> rdd.zipWithUniqueId().mapValues(uid => rdd.id * 1e9.toLong + uid)
>>
>> Just a hack ..
>>
>> On Wed, Aug 27, 2014 at 2:59 PM, Soumitra Kumar
>> <ku...@gmail.com> wrote:
>> > So, I guess zipWithUniqueId will be similar.
>> >
>> > Is there a way to get unique index?
>> >
>> >
>> > On Wed, Aug 27, 2014 at 2:39 PM, Xiangrui Meng <me...@gmail.com> wrote:
>> >>
>> >> No. The indices start at 0 for every RDD. -Xiangrui
>> >>
>> >> On Wed, Aug 27, 2014 at 2:37 PM, Soumitra Kumar
>> >> <ku...@gmail.com> wrote:
>> >> > Hello,
>> >> >
>> >> > If I do:
>> >> >
>> >> > DStream transform {
>> >> >     rdd.zipWithIndex.map {
>> >> >
>> >> >         Is the index guaranteed to be unique across all RDDs here?
>> >> >
>> >> > }
>> >> > }
>> >> >
>> >> > Thanks,
>> >> > -Soumitra.
>> >
>> >
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Spark Streaming: DStream - zipWithIndex

Posted by Soumitra Kumar <ku...@gmail.com>.

Thanks.

Just to double check, rdd.id would be unique for a batch in a DStream?


On Wed, Aug 27, 2014 at 3:04 PM, Xiangrui Meng <me...@gmail.com> wrote:

> You can use RDD id as the seed, which is unique in the same spark
> context. Suppose none of the RDDs would contain more than 1 billion
> records. Then you can use
>
> rdd.zipWithUniqueId().mapValues(uid => rdd.id * 1e9.toLong + uid)
>
> Just a hack ..
>
> On Wed, Aug 27, 2014 at 2:59 PM, Soumitra Kumar
> <ku...@gmail.com> wrote:
> > So, I guess zipWithUniqueId will be similar.
> >
> > Is there a way to get unique index?
> >
> >
> > On Wed, Aug 27, 2014 at 2:39 PM, Xiangrui Meng <me...@gmail.com> wrote:
> >>
> >> No. The indices start at 0 for every RDD. -Xiangrui
> >>
> >> On Wed, Aug 27, 2014 at 2:37 PM, Soumitra Kumar
> >> <ku...@gmail.com> wrote:
> >> > Hello,
> >> >
> >> > If I do:
> >> >
> >> > DStream transform {
> >> >     rdd.zipWithIndex.map {
> >> >
> >> >         Is the index guaranteed to be unique across all RDDs here?
> >> >
> >> > }
> >> > }
> >> >
> >> > Thanks,
> >> > -Soumitra.
> >
> >
>

Re: Spark Streaming: DStream - zipWithIndex

Posted by Tathagata Das <ta...@gmail.com>.

But then if you want to generate ids that are unique across ALL the records
that you are going to see in a stream (which can be potentially infinite),
then you definitely need a number space larger than long :)

TD


On Thu, Aug 28, 2014 at 12:48 PM, Soumitra Kumar <ku...@gmail.com>
wrote:

> Yes, that is an option.
>
> I started with a function of batch time, and index to generate id as long.
> This may be faster than generating UUID, with added benefit of sorting
> based on time.
>
> ----- Original Message -----
> From: "Tathagata Das" <ta...@gmail.com>
> To: "Soumitra Kumar" <ku...@gmail.com>
> Cc: "Xiangrui Meng" <me...@gmail.com>, user@spark.apache.org
> Sent: Thursday, August 28, 2014 2:19:38 AM
> Subject: Re: Spark Streaming: DStream - zipWithIndex
>
>
> If just want arbitrary unique id attached to each record in a dstream (no
> ordering etc), then why not create generate and attach an UUID to each
> record?
>
>
>
>
>
> On Wed, Aug 27, 2014 at 4:18 PM, Soumitra Kumar < kumar.soumitra@gmail.com
> > wrote:
>
>
>
> I see a issue here.
>
>
> If rdd.id is 1000 then rdd.id * 1e9.toLong would be BIG.
>
>
> I wish there was DStream mapPartitionsWithIndex.
>
>
>
>
>
> On Wed, Aug 27, 2014 at 3:04 PM, Xiangrui Meng < mengxr@gmail.com > wrote:
>
>
> You can use RDD id as the seed, which is unique in the same spark
> context. Suppose none of the RDDs would contain more than 1 billion
> records. Then you can use
>
> rdd.zipWithUniqueId().mapValues(uid => rdd.id * 1e9.toLong + uid)
>
> Just a hack ..
>
> On Wed, Aug 27, 2014 at 2:59 PM, Soumitra Kumar
>
>
> < kumar.soumitra@gmail.com > wrote:
> > So, I guess zipWithUniqueId will be similar.
> >
> > Is there a way to get unique index?
> >
> >
> > On Wed, Aug 27, 2014 at 2:39 PM, Xiangrui Meng < mengxr@gmail.com >
> wrote:
> >>
> >> No. The indices start at 0 for every RDD. -Xiangrui
> >>
> >> On Wed, Aug 27, 2014 at 2:37 PM, Soumitra Kumar
> >> < kumar.soumitra@gmail.com > wrote:
> >> > Hello,
> >> >
> >> > If I do:
> >> >
> >> > DStream transform {
> >> > rdd.zipWithIndex.map {
> >> >
> >> > Is the index guaranteed to be unique across all RDDs here?
> >> >
> >> > }
> >> > }
> >> >
> >> > Thanks,
> >> > -Soumitra.
> >
> >
>
>
>

Re: Spark Streaming: DStream - zipWithIndex

Posted by Soumitra Kumar <ku...@gmail.com>.

Yes, that is an option.

I started with a function of batch time, and index to generate id as long. This may be faster than generating UUID, with added benefit of sorting based on time.

----- Original Message -----
From: "Tathagata Das" <ta...@gmail.com>
To: "Soumitra Kumar" <ku...@gmail.com>
Cc: "Xiangrui Meng" <me...@gmail.com>, user@spark.apache.org
Sent: Thursday, August 28, 2014 2:19:38 AM
Subject: Re: Spark Streaming: DStream - zipWithIndex

If just want arbitrary unique id attached to each record in a dstream (no ordering etc), then why not create generate and attach an UUID to each record? 

On Wed, Aug 27, 2014 at 4:18 PM, Soumitra Kumar < kumar.soumitra@gmail.com > wrote: 

I see a issue here. 

If rdd.id is 1000 then rdd.id * 1e9.toLong would be BIG. 

I wish there was DStream mapPartitionsWithIndex. 

On Wed, Aug 27, 2014 at 3:04 PM, Xiangrui Meng < mengxr@gmail.com > wrote: 

You can use RDD id as the seed, which is unique in the same spark 
context. Suppose none of the RDDs would contain more than 1 billion 
records. Then you can use 

rdd.zipWithUniqueId().mapValues(uid => rdd.id * 1e9.toLong + uid) 

Just a hack .. 

On Wed, Aug 27, 2014 at 2:59 PM, Soumitra Kumar 

< kumar.soumitra@gmail.com > wrote: 
> So, I guess zipWithUniqueId will be similar. 
> 
> Is there a way to get unique index? 
> 
> 
> On Wed, Aug 27, 2014 at 2:39 PM, Xiangrui Meng < mengxr@gmail.com > wrote: 
>> 
>> No. The indices start at 0 for every RDD. -Xiangrui 
>> 
>> On Wed, Aug 27, 2014 at 2:37 PM, Soumitra Kumar 
>> < kumar.soumitra@gmail.com > wrote: 
>> > Hello, 
>> > 
>> > If I do: 
>> > 
>> > DStream transform { 
>> > rdd.zipWithIndex.map { 
>> > 
>> > Is the index guaranteed to be unique across all RDDs here? 
>> > 
>> > } 
>> > } 
>> > 
>> > Thanks, 
>> > -Soumitra. 
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Spark Streaming: DStream - zipWithIndex

Posted by Tathagata Das <ta...@gmail.com>.

If just want arbitrary unique id attached to each record in a dstream (no
ordering etc), then why not create generate and attach an UUID to each
record?



On Wed, Aug 27, 2014 at 4:18 PM, Soumitra Kumar <ku...@gmail.com>
wrote:

> I see a issue here.
>
> If rdd.id is 1000 then rdd.id * 1e9.toLong would be BIG.
>
> I wish there was DStream mapPartitionsWithIndex.
>
>
> On Wed, Aug 27, 2014 at 3:04 PM, Xiangrui Meng <me...@gmail.com> wrote:
>
>> You can use RDD id as the seed, which is unique in the same spark
>> context. Suppose none of the RDDs would contain more than 1 billion
>> records. Then you can use
>>
>> rdd.zipWithUniqueId().mapValues(uid => rdd.id * 1e9.toLong + uid)
>>
>> Just a hack ..
>>
>> On Wed, Aug 27, 2014 at 2:59 PM, Soumitra Kumar
>> <ku...@gmail.com> wrote:
>> > So, I guess zipWithUniqueId will be similar.
>> >
>> > Is there a way to get unique index?
>> >
>> >
>> > On Wed, Aug 27, 2014 at 2:39 PM, Xiangrui Meng <me...@gmail.com>
>> wrote:
>> >>
>> >> No. The indices start at 0 for every RDD. -Xiangrui
>> >>
>> >> On Wed, Aug 27, 2014 at 2:37 PM, Soumitra Kumar
>> >> <ku...@gmail.com> wrote:
>> >> > Hello,
>> >> >
>> >> > If I do:
>> >> >
>> >> > DStream transform {
>> >> >     rdd.zipWithIndex.map {
>> >> >
>> >> >         Is the index guaranteed to be unique across all RDDs here?
>> >> >
>> >> > }
>> >> > }
>> >> >
>> >> > Thanks,
>> >> > -Soumitra.
>> >
>> >
>>
>
>

Re: Spark Streaming: DStream - zipWithIndex

Posted by Soumitra Kumar <ku...@gmail.com>.

I see a issue here.

If rdd.id is 1000 then rdd.id * 1e9.toLong would be BIG.

I wish there was DStream mapPartitionsWithIndex.


On Wed, Aug 27, 2014 at 3:04 PM, Xiangrui Meng <me...@gmail.com> wrote:

> You can use RDD id as the seed, which is unique in the same spark
> context. Suppose none of the RDDs would contain more than 1 billion
> records. Then you can use
>
> rdd.zipWithUniqueId().mapValues(uid => rdd.id * 1e9.toLong + uid)
>
> Just a hack ..
>
> On Wed, Aug 27, 2014 at 2:59 PM, Soumitra Kumar
> <ku...@gmail.com> wrote:
> > So, I guess zipWithUniqueId will be similar.
> >
> > Is there a way to get unique index?
> >
> >
> > On Wed, Aug 27, 2014 at 2:39 PM, Xiangrui Meng <me...@gmail.com> wrote:
> >>
> >> No. The indices start at 0 for every RDD. -Xiangrui
> >>
> >> On Wed, Aug 27, 2014 at 2:37 PM, Soumitra Kumar
> >> <ku...@gmail.com> wrote:
> >> > Hello,
> >> >
> >> > If I do:
> >> >
> >> > DStream transform {
> >> >     rdd.zipWithIndex.map {
> >> >
> >> >         Is the index guaranteed to be unique across all RDDs here?
> >> >
> >> > }
> >> > }
> >> >
> >> > Thanks,
> >> > -Soumitra.
> >
> >
>

Re: Spark Streaming: DStream - zipWithIndex

Posted by Xiangrui Meng <me...@gmail.com>.

You can use RDD id as the seed, which is unique in the same spark
context. Suppose none of the RDDs would contain more than 1 billion
records. Then you can use

rdd.zipWithUniqueId().mapValues(uid => rdd.id * 1e9.toLong + uid)

Just a hack ..

On Wed, Aug 27, 2014 at 2:59 PM, Soumitra Kumar
<ku...@gmail.com> wrote:
> So, I guess zipWithUniqueId will be similar.
>
> Is there a way to get unique index?
>
>
> On Wed, Aug 27, 2014 at 2:39 PM, Xiangrui Meng <me...@gmail.com> wrote:
>>
>> No. The indices start at 0 for every RDD. -Xiangrui
>>
>> On Wed, Aug 27, 2014 at 2:37 PM, Soumitra Kumar
>> <ku...@gmail.com> wrote:
>> > Hello,
>> >
>> > If I do:
>> >
>> > DStream transform {
>> >     rdd.zipWithIndex.map {
>> >
>> >         Is the index guaranteed to be unique across all RDDs here?
>> >
>> > }
>> > }
>> >
>> > Thanks,
>> > -Soumitra.
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Spark Streaming: DStream - zipWithIndex

Posted by Soumitra Kumar <ku...@gmail.com>.

So, I guess zipWithUniqueId will be similar.

Is there a way to get unique index?

On Wed, Aug 27, 2014 at 2:39 PM, Xiangrui Meng <me...@gmail.com> wrote:

> No. The indices start at 0 for every RDD. -Xiangrui
>
> On Wed, Aug 27, 2014 at 2:37 PM, Soumitra Kumar
> <ku...@gmail.com> wrote:
> > Hello,
> >
> > If I do:
> >
> > DStream transform {
> >     rdd.zipWithIndex.map {
> >
> >         Is the index guaranteed to be unique across all RDDs here?
> >
> > }
> > }
> >
> > Thanks,
> > -Soumitra.
>

Re: Spark Streaming: DStream - zipWithIndex

Posted by Xiangrui Meng <me...@gmail.com>.

No. The indices start at 0 for every RDD. -Xiangrui

On Wed, Aug 27, 2014 at 2:37 PM, Soumitra Kumar
<ku...@gmail.com> wrote:
> Hello,
>
> If I do:
>
> DStream transform {
>     rdd.zipWithIndex.map {
>
>         Is the index guaranteed to be unique across all RDDs here?
>
> }
> }
>
> Thanks,
> -Soumitra.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org