[jira] [Commented] (HIVE-7856) Enable parallelism in Reduce Side Join [Spark Branch]

["Szehon Ho (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14107577#comment-14107577 ] 

Szehon Ho commented on HIVE-7856:
---------------------------------

Some preliminary research on how this would work, we would use the SPARK-2978 group+sort in the shuffle phase, and we can use the HashPartitioner to decide the partition.  Today in MapReduce we are using the default partitioner (hash partitioner), as the HiveKey has a pre-computed hash-code field, that directs it to the right HiveReduce function. 

One issue is that HiveBaseFunctionResultList clones the HiveKey as BytesWritable during mapper-output generation, losing the hashcode.  We need to make a change there to preserve the hashcode, for this to work.

> Enable parallelism in Reduce Side Join [Spark Branch]
> -----------------------------------------------------
>
>                 Key: HIVE-7856
>                 URL: https://issues.apache.org/jira/browse/HIVE-7856
>             Project: Hive
>          Issue Type: New Feature
>          Components: Spark
>            Reporter: Szehon Ho
>
> This is dependent on new transformation to be provided by SPARK-2978, see parent JIRA for details.



--
This message was sent by Atlassian JIRA
(v6.2#6252)