[jira] [Work started] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible

["Jesus Camacho Rodriguez (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on HIVE-13750 started by Jesus Camacho Rodriguez.
------------------------------------------------------
> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible
> ------------------------------------------------------------------------------------------
>
>                 Key: HIVE-13750
>                 URL: https://issues.apache.org/jira/browse/HIVE-13750
>             Project: Hive
>          Issue Type: Improvement
>          Components: Physical Optimizer
>    Affects Versions: 2.1.0
>            Reporter: Jesus Camacho Rodriguez
>            Assignee: Jesus Camacho Rodriguez
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO) unconditionally adds an extra shuffle stage. If sort columns of previous shuffle and partitioning columns of table match, reduce sink deduplication optimizer removes extra shuffle stage, thus bringing down overhead to zero. However, if they don’t match, we end up doing extra shuffle. This can be improved since we can add table partition columns as a sort columns on earlier shuffle and avoid this extra shuffle. This ensures that in cases query already has a shuffle stage, we are not shuffling data again. 
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)