You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Manoj Awasthi <aw...@gmail.com> on 2014/03/06 13:09:08 UTC
special case of custom partitioning
Hi All,
I have a three machine cluster. I have two RDDs each consisting of (K,V)
pairs. RDDs have just three keys 'a', 'b' and 'c'.
// list1 - List(('a',1), ('b',2), ....
val rdd1 = sc.parallelize(list1).groupByKey(new HashPartitioner(3))
// list2 - List(('a',2), ('b',7), ....
val rdd2 = sc.parallelize(list2).groupByKey(new HashPartitioner(3))
By using a HashPartitioner with 3 partitions I can achieve that each of the
keys ('a', 'b' and 'c') in each RDD gets partitioned on different machines
on cluster (based on the hashCode).
Problem is that I cannot deterministically do the same allocation for
second RDD? (all 'a's from rdd2 going to the same machine where 'a's from
first RDD went to).
Is there a way to achieve this?
Manoj
Re: special case of custom partitioning
Posted by Manoj Awasthi <aw...@gmail.com>.
Thanks Mayur - based on the doc-comments in source looks like this will
work for the case. I will confirm.
----
the dreamers of the day are dangerous men, for they may act their dream
with open eyes, and make it possible
On Fri, Mar 7, 2014 at 2:21 AM, Mayur Rustagi <ma...@gmail.com>wrote:
> How about PartitionerAwareUnionRDD?
>
> Regards
> Mayur
>
> Mayur Rustagi
> Ph: +1 (760) 203 3257
> http://www.sigmoidanalytics.com
> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>
>
>
> On Thu, Mar 6, 2014 at 9:42 AM, Evan Chan <ev...@ooyala.com> wrote:
>
> > I would love to hear the answer to this as well.
> >
> > On Thu, Mar 6, 2014 at 4:09 AM, Manoj Awasthi <aw...@gmail.com>
> > wrote:
> > > Hi All,
> > >
> > >
> > > I have a three machine cluster. I have two RDDs each consisting of
> (K,V)
> > > pairs. RDDs have just three keys 'a', 'b' and 'c'.
> > >
> > > // list1 - List(('a',1), ('b',2), ....
> > > val rdd1 = sc.parallelize(list1).groupByKey(new HashPartitioner(3))
> > >
> > > // list2 - List(('a',2), ('b',7), ....
> > > val rdd2 = sc.parallelize(list2).groupByKey(new HashPartitioner(3))
> > >
> > > By using a HashPartitioner with 3 partitions I can achieve that each of
> > the
> > > keys ('a', 'b' and 'c') in each RDD gets partitioned on different
> > machines
> > > on cluster (based on the hashCode).
> > >
> > > Problem is that I cannot deterministically do the same allocation for
> > > second RDD? (all 'a's from rdd2 going to the same machine where 'a's
> from
> > > first RDD went to).
> > >
> > > Is there a way to achieve this?
> > >
> > > Manoj
> >
> >
> >
> > --
> > --
> > Evan Chan
> > Staff Engineer
> > ev@ooyala.com |
> >
>
Re: special case of custom partitioning
Posted by Mayur Rustagi <ma...@gmail.com>.
How about PartitionerAwareUnionRDD?
Regards
Mayur
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>
On Thu, Mar 6, 2014 at 9:42 AM, Evan Chan <ev...@ooyala.com> wrote:
> I would love to hear the answer to this as well.
>
> On Thu, Mar 6, 2014 at 4:09 AM, Manoj Awasthi <aw...@gmail.com>
> wrote:
> > Hi All,
> >
> >
> > I have a three machine cluster. I have two RDDs each consisting of (K,V)
> > pairs. RDDs have just three keys 'a', 'b' and 'c'.
> >
> > // list1 - List(('a',1), ('b',2), ....
> > val rdd1 = sc.parallelize(list1).groupByKey(new HashPartitioner(3))
> >
> > // list2 - List(('a',2), ('b',7), ....
> > val rdd2 = sc.parallelize(list2).groupByKey(new HashPartitioner(3))
> >
> > By using a HashPartitioner with 3 partitions I can achieve that each of
> the
> > keys ('a', 'b' and 'c') in each RDD gets partitioned on different
> machines
> > on cluster (based on the hashCode).
> >
> > Problem is that I cannot deterministically do the same allocation for
> > second RDD? (all 'a's from rdd2 going to the same machine where 'a's from
> > first RDD went to).
> >
> > Is there a way to achieve this?
> >
> > Manoj
>
>
>
> --
> --
> Evan Chan
> Staff Engineer
> ev@ooyala.com |
>
Re: special case of custom partitioning
Posted by Evan Chan <ev...@ooyala.com>.
I would love to hear the answer to this as well.
On Thu, Mar 6, 2014 at 4:09 AM, Manoj Awasthi <aw...@gmail.com> wrote:
> Hi All,
>
>
> I have a three machine cluster. I have two RDDs each consisting of (K,V)
> pairs. RDDs have just three keys 'a', 'b' and 'c'.
>
> // list1 - List(('a',1), ('b',2), ....
> val rdd1 = sc.parallelize(list1).groupByKey(new HashPartitioner(3))
>
> // list2 - List(('a',2), ('b',7), ....
> val rdd2 = sc.parallelize(list2).groupByKey(new HashPartitioner(3))
>
> By using a HashPartitioner with 3 partitions I can achieve that each of the
> keys ('a', 'b' and 'c') in each RDD gets partitioned on different machines
> on cluster (based on the hashCode).
>
> Problem is that I cannot deterministically do the same allocation for
> second RDD? (all 'a's from rdd2 going to the same machine where 'a's from
> first RDD went to).
>
> Is there a way to achieve this?
>
> Manoj
--
--
Evan Chan
Staff Engineer
ev@ooyala.com |