[jira] [Commented] (PIG-4066) An optimization for ROLLUP operation in Pig

["Daniel Dai (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555201#comment-14555201 ] 

Daniel Dai commented on PIG-4066:
---------------------------------

Looking at the patch while trying to document it. The idea is good and simple, however, there are couple of issues in the implementation:
1. Some basic queries does not work, eg: "cubed_and_rolled = CUBE salesinp BY CUBE(product,year), ROLLUP(region, state, city) pivot 1;"
2. Even if there is no "pivot" keyword, the implementation still using the new Pivot code
3. All script will go through RollupHIIOptimizer, it's on by default. Both #2 and #3 makes it impossible to just make it experimental feature and ship
4. The logic of RollupHII should be wrapped into the new operator, not necessary propagate to cogroup/UserFuncExpression, etc
5. There are a lot of redundant code needs to be cleaned up
6. Not a show stop but would like to port it to Tez as well

I already did quite a few cleanup. Since it will touch a majority part of the original patch, to make the commit history less confusing, I'd like to rollback the patch completely first and then redo it.

> An optimization for ROLLUP operation in Pig
> -------------------------------------------
>
>                 Key: PIG-4066
>                 URL: https://issues.apache.org/jira/browse/PIG-4066
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Quang-Nhat HOANG-XUAN
>            Assignee: Quang-Nhat HOANG-XUAN
>              Labels: hybrid-irg, optimization, rollup
>             Fix For: 0.15.0
>
>         Attachments: Current Rollup vs Our Rollup.jpg, PIG-4066.2.patch, PIG-4066.3.patch, PIG-4066.4.patch, PIG-4066.5.patch, PIG-4066.patch, TechnicalNotes.2.pdf, TechnicalNotes.pdf, UserGuide.pdf
>
>
> This patch aims at addressing the current limitation of the ROLLUP operator in PIG: most of the work is done in the Map phase of the underlying MapReduce job to generate all possible intermediate keys that the reducer use to aggregate and produce the ROLLUP output. Based on our previous work: “Duy-Hung Phan, Matteo Dell’Amico, Pietro Michiardi: On the design space of MapReduce ROLLUP aggregates” (http://www.eurecom.fr/en/publication/4212/download/rs-publi-4212_2.pdf), we show that the design space for a ROLLUP implementation allows for a different approach (in-reducer grouping, IRG), in which less work is done in the Map phase and the grouping is done in the Reduce phase. This patch presents the most efficient implementation we designed (Hybrid IRG), which allows defining a parameter to balance between parallelism (in the reducers) and communication cost.
> This patch contains the following features:
> 1. The new ROLLUP approach: IRG, Hybrid IRG.
> 2. The PIVOT clause in CUBE operators.
> 3. Test cases.
> The new syntax to use our ROLLUP approach:
> alias = CUBE rel BY { CUBE col_ref | ROLLUP col_ref [PIVOT pivot_value]} [, { CUBE col_ref | ROLLUP col_ref [PIVOT pivot_value]}...]
> In case there is multiple ROLLUP operator in one CUBE clause, the last ROLLUP operator will be executed with our approach (IRG, Hybrid IRG) while the remaining ROLLUP ahead will be executed with the default approach.
> We have already made some experiments for comparison between our ROLLUP implementation and the current ROLLUP. More information can be found at here: http://hxquangnhat.github.io/PIG-ROLLUP-H2IRG/
> Patch can be reviewed at here: https://reviews.apache.org/r/23804/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)