You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Steve Lewis <lo...@gmail.com> on 2014/09/02 23:27:52 UTC

I am looking for a Java sample of a Partitioner

Assume say JavaWord count
I call the equivalent of a Mapper
 JavaPairRDD<String, Integer> ones = words.mapToPair(,,,

Now right here I want to guarantee that each word starting with a
particular letter is processed in a specific partition - (Don't tell me
this is a dumb idea - I know that but in a Hadoop code a custom partitioner
is often important and I don't want to explain the real case)

I have no idea how ones would implement mapToPartition but I want emulate
Hadoop with a custom partition and keySort order


    JavaPairRDD<String, Integer> counts = ones.reduceByKey(...