You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by dash <bs...@nd.edu> on 2015/02/05 00:52:44 UTC

New combination-like RDD based on two RDDs

Hey Spark gurus! Sorry for the confusing title. I do not know the exactly
description of my problem, if you know please tell me so I can change it :-)

Say I have two RDDs right now, and they are

val rdd1 = sc.parallelize(List((1,(3)), (2,(5)), (3,(6))))
val rdd2 = sc.parallelize(List((2,(1)), (2,(3)), (3,(9))))

I want combine rdd1 and rdd2 to get rdd3 which looks like

List((1,(3)), (2,(5,1)), (2,(5,3)), (3, (6,9)))

The order in _._2 does not matter, so you can treat it as a Set.

I tried to use zip, but since there is no guarantee that the length of rdd1
and rdd2 will be the same I do not know if it is doable.

Also I looked into PairedRDD, some people use union operation on two RDDs
and then apply a map function on it. Since I want all combinations according
to _._1, I do not know how to achieve it by union and map.

Thanks in advance!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/New-combination-like-RDD-based-on-two-RDDs-tp21508.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: New combination-like RDD based on two RDDs

Posted by Jim Green <op...@gmail.com>.
You should use join:
val rdd1 = sc.parallelize(List((1,(3)), (2,(5)), (3,(6))))
val rdd2 = sc.parallelize(List((2,(1)), (2,(3)), (3,(9))))

rdd1.join(rdd2).collect
res0: Array[(Int, (Int, Int))] = Array((2,(5,1)), (2,(5,3)), (3,(6,9)))

Please see my cheat sheet at
* 3.14 join(otherDataset, [numTasks])    *
http://www.openkb.info/2015/01/scala-on-spark-cheatsheet.html


On Wed, Feb 4, 2015 at 3:52 PM, dash <bs...@nd.edu> wrote:

> Hey Spark gurus! Sorry for the confusing title. I do not know the exactly
> description of my problem, if you know please tell me so I can change it
> :-)
>
> Say I have two RDDs right now, and they are
>
> val rdd1 = sc.parallelize(List((1,(3)), (2,(5)), (3,(6))))
> val rdd2 = sc.parallelize(List((2,(1)), (2,(3)), (3,(9))))
>
> I want combine rdd1 and rdd2 to get rdd3 which looks like
>
> List((1,(3)), (2,(5,1)), (2,(5,3)), (3, (6,9)))
>
> The order in _._2 does not matter, so you can treat it as a Set.
>
> I tried to use zip, but since there is no guarantee that the length of rdd1
> and rdd2 will be the same I do not know if it is doable.
>
> Also I looked into PairedRDD, some people use union operation on two RDDs
> and then apply a map function on it. Since I want all combinations
> according
> to _._1, I do not know how to achieve it by union and map.
>
> Thanks in advance!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/New-combination-like-RDD-based-on-two-RDDs-tp21508.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>


-- 
Thanks,
www.openkb.info
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)

Re: New combination-like RDD based on two RDDs

Posted by dash <bs...@nd.edu>.
Problem solved. A simple join will do the work

val prefix = new PairRDDFunctions[Int, Set[Int]](sc.parallelize(List((9,
Set(4)), (1,Set(3)), (2,Set(5)), (2,Set(4)))))

val suffix = sc.parallelize(List((1, Set(1)), (2, Set(6)), (2, Set(5)), (2,
Set(7))))

prefix.join(suffix).collect().foreach(println)





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/New-combination-like-RDD-based-on-two-RDDs-tp21508p21511.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org