You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Ashutosh Chauhan (JIRA)" <ji...@apache.org> on 2010/06/10 19:24:14 UTC
[jira] Created: (PIG-1446) OOME in a query having a bincond in the
inner plan of a Foreach.
OOME in a query having a bincond in the inner plan of a Foreach.
----------------------------------------------------------------
Key: PIG-1446
URL: https://issues.apache.org/jira/browse/PIG-1446
Project: Pig
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Ashutosh Chauhan
This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-1446) OOME in a query having a bincond in the
inner plan of a Foreach.
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan reassigned PIG-1446:
-------------------------------------
Assignee: Ashutosh Chauhan
> OOME in a query having a bincond in the inner plan of a Foreach.
> ----------------------------------------------------------------
>
> Key: PIG-1446
> URL: https://issues.apache.org/jira/browse/PIG-1446
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Attachments: pig-1446.patch
>
>
> This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1446) OOME in a query having a bincond in
the inner plan of a Foreach.
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877540#action_12877540 ]
Daniel Dai commented on PIG-1446:
---------------------------------
+1. Please commit after test and hudson pass.
> OOME in a query having a bincond in the inner plan of a Foreach.
> ----------------------------------------------------------------
>
> Key: PIG-1446
> URL: https://issues.apache.org/jira/browse/PIG-1446
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Attachments: pig-1446.patch
>
>
> This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1446) OOME in a query having a bincond in the
inner plan of a Foreach.
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1446:
----------------------------------
Status: Resolved (was: Patch Available)
Fix Version/s: 0.8.0
0.7.0
Resolution: Fixed
As usual, hudson is not responding. I manually ran all the unit tests, all of them passed. Committed to both trunk and 0.7
> OOME in a query having a bincond in the inner plan of a Foreach.
> ----------------------------------------------------------------
>
> Key: PIG-1446
> URL: https://issues.apache.org/jira/browse/PIG-1446
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Fix For: 0.8.0, 0.7.0
>
> Attachments: pig-1446.patch
>
>
> This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1446) OOME in a query having a bincond in the
inner plan of a Foreach.
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1446:
----------------------------------
Status: Patch Available (was: Open)
> OOME in a query having a bincond in the inner plan of a Foreach.
> ----------------------------------------------------------------
>
> Key: PIG-1446
> URL: https://issues.apache.org/jira/browse/PIG-1446
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Attachments: pig-1446.patch
>
>
> This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1446) OOME in a query having a bincond in the
inner plan of a Foreach.
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-1446:
----------------------------------
Attachment: pig-1446.patch
Sequence of event is as follows:
1) MultiQuery optimizer combined 30 group-bys in one reducer. So, there are 30 pipelines in a reducer.
2) Each of these group-by has a ForEach after them.
3) ForEach has a bincond in its own plan.
4) Group-by resulted in large bags (10s of million of records).
5) Tuple containing group and bag is attached to the roots of inner plan of FE.
6) FE pulled the tuples through its leaves.
7) Due to short-circuiting in bin-cond, one branch of the plan is never pulled resulting in stray reference of bag which actually was not needed.
8) Due to MQ optimized 30 group-bys, we had many such bags now hanging in there, eating up all the memory.
Fix: Detach tuples from the roots once you are done in FE.
> OOME in a query having a bincond in the inner plan of a Foreach.
> ----------------------------------------------------------------
>
> Key: PIG-1446
> URL: https://issues.apache.org/jira/browse/PIG-1446
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Attachments: pig-1446.patch
>
>
> This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.