[jira] [Updated] (PIG-2829) Use partial aggregation more aggresively

["Jie Li (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jie Li updated PIG-2829:
------------------------

    Attachment: tpch-10G.png
                pigmix-10G.png

Attached some benchmark results of a group-by aggregation on different dataset (TPC-H and Pigmix) with different selectivity and with combiner/PartialAgg turned on/off respectively. 

'none' means not using combiner or PartialAgg. 'combiner' means only using the combiner. 'hash' means only using the PartialAgg (with some hack). 'hash+combiner' means enabling PartialAgg. For the latter two we configures the minimum reduction to 1 so PartialAgg is never auto-disabled (otherwise it'd be auto-disabled in all cases currently). For TPC-H We also run Hive with default settings for reference.

The titles above each chart show the number of input records and output records of each query, for example, "60M -> 4 reduction" means there are 60 million input records and four output records. For both dataset we use 10GB data, and ran on a single machine, which is ok here as we are comparing PartialAgg with combiner so the network doesn't matter much here. 

>From the results we can observe:

1) PartialAgg is more efficient than the combiner, which is as expected and should be leveraged;
2) the combiner is unnecessary when PartialAgg is used;
3) the PartialAgg/combiner overhead can be significant if the data reduction rate is low.

                
> Use partial aggregation more aggresively
> ----------------------------------------
>
>                 Key: PIG-2829
>                 URL: https://issues.apache.org/jira/browse/PIG-2829
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: 0.10.0
>            Reporter: Jie Li
>         Attachments: pigmix-10G.png, tpch-10G.png
>
>
> Partial aggregation (Hash Aggregation, aka in-map combiner) is a new feature in Pig 0.10 that will perform aggregation within map function. The main advantage against combiner is it avoids de/serializing and sorting the data, and it can auto disable itself if the data reduction rate is low. Currently it's disabled by default.
> To leverage the power of PartialAgg more aggressively, several things need to be revisited:
> 1. The threshold of auto-disabling. Currently each mapper looks at first 1k (hard-coded) records to see if there's enough data size reduction (defaults to 10x, configurable). The check would happen earlier if the hash table gets full before processing the 1k records (hash table size is controlled by pig.cachedbag.memusage). We might want to relax these thresholds.
> 2. Dependency on the combiner. Currently the PartialAgg won't work without a combiner following it, so we need to provide separate options to enable each independently. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira