[jira] Updated: (PIG-1446) OOME in a query having a bincond in the inner plan of a Foreach.

["Ashutosh Chauhan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ashutosh Chauhan updated PIG-1446:
----------------------------------

    Attachment: pig-1446.patch

Sequence of event is as follows:
1) MultiQuery optimizer combined 30 group-bys in one reducer. So, there are 30 pipelines in a reducer.
2) Each of these group-by has a ForEach after them.
3) ForEach has a bincond in its own plan.
4) Group-by resulted in large bags (10s of million of records).
5) Tuple containing group and bag is attached to the roots of inner plan of FE.
6) FE pulled the tuples through its leaves.
7) Due to short-circuiting in bin-cond, one branch of the plan is never pulled resulting in stray reference of bag which actually was not needed.
8) Due to MQ optimized 30 group-bys, we had many such bags now hanging in there, eating up all the memory.

Fix: Detach tuples from the roots once you are done in FE.

> OOME in a query having a bincond in the inner plan of a Foreach.
> ----------------------------------------------------------------
>
>                 Key: PIG-1446
>                 URL: https://issues.apache.org/jira/browse/PIG-1446
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.7.0
>            Reporter: Ashutosh Chauhan
>         Attachments: pig-1446.patch
>
>
> This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.