[jira] [Updated] (PIG-3781) Fix TestTezCompiler#testReplicatedJoinInReducer

["Cheolsoo Park (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Cheolsoo Park updated PIG-3781:
-------------------------------

    Attachment: PIG-3781-1.patch

In the attached patch, I changed the condition in Tez CombinerOptimizer for detecting co-group as follows-

From:
{code}
        if (predecessors.size()>1) {
{code}
To:
{code}
        if (to.isCogroup()) {
{code}

One corner case that the old condition fails is when group-by + algebraic udf are followed by join (e.g. TestTezCompiler.testReplicatedJoinInReducer). In this case, we want to add combiner to the output edge of group-by. In MR, this cannot happen because join never has more than one inputs, so checking the number of predecessors was fine.

Now I am explicitly checking whether the reduce vertex is co-group or not. I also added a new test case to TestTezCompiler to verify that the combiner is disabled for co-group.



> Fix TestTezCompiler#testReplicatedJoinInReducer
> -----------------------------------------------
>
>                 Key: PIG-3781
>                 URL: https://issues.apache.org/jira/browse/PIG-3781
>             Project: Pig
>          Issue Type: Sub-task
>          Components: tez
>    Affects Versions: tez-branch
>            Reporter: Cheolsoo Park
>            Assignee: Cheolsoo Park
>             Fix For: tez-branch
>
>         Attachments: PIG-3781-1.patch
>
>
> TestTezCompiler#testReplicatedJoinInReducer is broken because of PIG-3759. We need to fix it.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)