You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Travis Woodruff (JIRA)" <ji...@apache.org> on 2014/06/09 22:27:02 UTC
[jira] [Commented] (PIG-4002) Disable combiner when map-side
aggregation is used
[ https://issues.apache.org/jira/browse/PIG-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14025691#comment-14025691 ]
Travis Woodruff commented on PIG-4002:
--------------------------------------
The drawback of doing this is that it removes the combiner plan completely, so if in-memory aggregation is disabled at execution time (by POPartialAgg.checkSizeReduction()), the combiner won't be used. One could argue that this is acceptable since if in-memory aggregation didn't provide a substantial reduction, the combiner wouldn't either.
> Disable combiner when map-side aggregation is used
> --------------------------------------------------
>
> Key: PIG-4002
> URL: https://issues.apache.org/jira/browse/PIG-4002
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.13.0
> Reporter: Travis Woodruff
> Assignee: Travis Woodruff
> Priority: Minor
> Attachments: PIG-4002-1.patch
>
>
> This may be controversial, so I'd like others' opinions on this.
> It is not currently possible to disable the combiner and use map-side aggregation at the same time. This is a problematic since map-side aggregation effectively combines in the mapper, so running the combiner adds expensive combiner execution (combiner requires deserialization & reserialization) for little to no value.
> PIG-2829 had a patch to disable the combiner when map-side aggregation is used (along with some other changes). This was never integrated because the map-side aggregation code was redone while this was in progress.
--
This message was sent by Atlassian JIRA
(v6.2#6252)