You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Mohit Sabharwal (JIRA)" <ji...@apache.org> on 2015/05/21 03:55:59 UTC

[jira] [Created] (PIG-4565) Support custom MR partitioners for Spark engine

Mohit Sabharwal created PIG-4565:
------------------------------------

             Summary: Support custom MR partitioners for Spark engine 
                 Key: PIG-4565
                 URL: https://issues.apache.org/jira/browse/PIG-4565
             Project: Pig
          Issue Type: Sub-task
          Components: spark
    Affects Versions: spark-branch
            Reporter: Mohit Sabharwal
            Assignee: Mohit Sabharwal
             Fix For: spark-branch


Shuffle operations like DISTINCT, GROUP, JOIN, CROSS allow custom MR partitioners to be specified.

Example:
{code}
B = GROUP A BY $0 PARTITION BY org.apache.pig.test.utils.SimpleCustomPartitioner PARALLEL 2;

public class SimpleCustomPartitioner extends Partitioner <PigNullableWritable, Writable> { 
     //@Override 
    public int getPartition(PigNullableWritable key, Writable value, int numPartitions) { 
        if(key.getValueAsPigType() instanceof Integer) { 
            int ret = (((Integer)key.getValueAsPigType()).intValue() % numPartitions); 
            return ret; 
       } 
       else { 
            return (key.hashCode()) % numPartitions; 
        } 
    } 
}
{code}

Since Spark's shuffle APIs takes a different parititioner class (org.apache.spark.Partitioner) compared to MapReduce (org.apache.hadoop.mapreduce.Partitioner), we need to wrap custom partitioners written for MapReduce inside a Spark Partitioner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)