You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Walrus theCat <wa...@gmail.com> on 2014/03/26 19:11:30 UTC

interleave partitions?

Hi,

I want to do something like this:

rdd3 = rdd1.coalesce(N).partitions.zip(rdd2.coalesce(N).partitions)

I realize the above will get me something like Array[(partition,partition)].

I hope you see what I'm going for here -- any tips on how to accomplish
this?

Thanks

Re: interleave partitions?

Posted by Walrus theCat <wa...@gmail.com>.

Answering my own question here.  This may not be efficient, but this is
what I came up with:

rdd1.coalesce(N).glom.zip(rdd2.coalesce(N).glom).map { case(x,y) => x++y}

On Wed, Mar 26, 2014 at 11:11 AM, Walrus theCat <wa...@gmail.com>wrote:

> Hi,
>
> I want to do something like this:
>
> rdd3 = rdd1.coalesce(N).partitions.zip(rdd2.coalesce(N).partitions)
>
> I realize the above will get me something like
> Array[(partition,partition)].
>
> I hope you see what I'm going for here -- any tips on how to accomplish
> this?
>
> Thanks
>