You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Tom <th...@gmail.com> on 2014/09/16 15:42:24 UTC
Reduce Tuple2 to Tuple2>>
>From my map function I create Tuple2<Integer, Integer> pairs. Now I want to
reduce them, and get something like Tuple2<Integer, List<Integer>>.
The only way I found to do this was by treating all variables as String, and
in the reduceByKey do
/return a._2 + "," + b._2/ //in which both are numeric values saved in a
String
After which I do a Arrays.asList(string.split(",")) in mapValues. This
leaves me with <String, List<Integer>>. So now I am looking for either
- A function with which I can transform <String, List<Integer>> to
<Integer, List<Integer>>
or
- A way to reduce Tuple2<Integer, Integer> into a Tuple2<Integer,
List<Integer>> in the reduceByKey function so that I can use Integers all
the way
Of course option two would have preferences.
Thanks!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Reduce-Tuple2-Integer-Integer-to-Tuple2-Integer-List-Integer-tp14361.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Reduce Tuple2 to Tuple2>>
Posted by Sean Owen <so...@cloudera.com>.
If you mean you have (key,value) pairs, and want pairs with key, and
all values for that key, then you're looking for groupByKey
On Tue, Sep 16, 2014 at 2:42 PM, Tom <th...@gmail.com> wrote:
> From my map function I create Tuple2<Integer, Integer> pairs. Now I want to
> reduce them, and get something like Tuple2<Integer, List<Integer>>.
>
> The only way I found to do this was by treating all variables as String, and
> in the reduceByKey do
> /return a._2 + "," + b._2/ //in which both are numeric values saved in a
> String
> After which I do a Arrays.asList(string.split(",")) in mapValues. This
> leaves me with <String, List<Integer>>. So now I am looking for either
> - A function with which I can transform <String, List<Integer>> to
> <Integer, List<Integer>>
> or
> - A way to reduce Tuple2<Integer, Integer> into a Tuple2<Integer,
> List<Integer>> in the reduceByKey function so that I can use Integers all
> the way
>
> Of course option two would have preferences.
>
> Thanks!
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Reduce-Tuple2-Integer-Integer-to-Tuple2-Integer-List-Integer-tp14361.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org