[jira] Commented: (HIVE-1467) dynamic partitioning should cluster by partitions

["Ning Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905700#action_12905700 ] 

Ning Zhang commented on HIVE-1467:
----------------------------------

As discussed with Joydeep and Ashish, it seems we should use the "distribute by" mechanism rather than "cluster by" to avoid sorting at the reducer side. The difference between them is "distribute by" only have MapReduce partition columns set to be the Dyanmic partition columns, and "cluster by" will additionally set "key columns" as the dynamic partition columns as well.

So I think we can use 2 mode of reducer-side DP with tradeoffs:
  -- distribute by mode: no sorting but reducers have to keep all files open during DP insert. Good choice when there are large amount of data passed from mappers to reducers.
  -- cluster by mode: sorting by the DP columns, but we can close a DP file once FileSinkOperator sees a dfferent DP column value. Good choice when total data size is not that large but there are large number of DPs generated.  

> dynamic partitioning should cluster by partitions
> -------------------------------------------------
>
>                 Key: HIVE-1467
>                 URL: https://issues.apache.org/jira/browse/HIVE-1467
>             Project: Hadoop Hive
>          Issue Type: Improvement
>            Reporter: Joydeep Sen Sarma
>            Assignee: Namit Jain
>
> (based on internal discussion with Ning). Dynamic partitioning should offer a mode where it clusters data by partition before writing out to each partition. This will reduce number of files. Details:
> 1. always use reducer stage
> 2. mapper sends to reducer based on partitioning column. ie. reducer = f(partition-cols)
> 3. f() can be made somewhat smart to:
>    a. spread large partitions across multiple reducers - each mapper can maintain row count seen per partition - and then apply (whenever it sees a new row for a partition): 
>        * reducer = (row count / 64k) % numReducers 
>        Small partitions always go to one reducer. the larger the partition, the more the reducers. this prevents one reducer becoming bottleneck writing out one partition
>    b. this still leaves the issue of very large number of splits. (64K rows from 10K mappers is pretty large). for this one can apply one slight modification:
>        * reducer = (mapper-id/1024 + row-count/64k) % numReducers
>    ie. - the first 1000 mappers always send the first 64K rows for one partition to the same reducer. the next 1000 send it to the next one. and so on.
> the constants 1024 and 64k are used just as an example. i don't know what the right numbers are. it's also clear that this is a case where we need hadoop to do only partitioning (and no sorting). this will be a useful feature to have in hadoop. that will reduce the overhead due to reducers.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.