You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2008/08/07 00:54:44 UTC
[jira] Created: (PIG-362) Limit can not push in front of ForEach
with flatten
Limit can not push in front of ForEach with flatten
---------------------------------------------------
Key: PIG-362
URL: https://issues.apache.org/jira/browse/PIG-362
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: types_branch
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: types_branch
Currently logical optimizer will push Limit in front of ForEach with flatten. It is based on the assumption that ForEach with a flatten always increase the number of records. However, this is a false assumption. In the case that there is empty bags inside input tuple, the number of output records can be 0, thus less than input records.
We have no way to know whether there is an empty bag in the input at optimization time. So the only solution is not to push Limit in front of ForEach with flatten
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-362) Limit can not push in front of ForEach
with flatten
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-362:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
patch committed. thanks daniel for contributing!
> Limit can not push in front of ForEach with flatten
> ---------------------------------------------------
>
> Key: PIG-362
> URL: https://issues.apache.org/jira/browse/PIG-362
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: types_branch
>
> Attachments: PIG-362-2.patch, PIG-362.patch
>
>
> Currently logical optimizer will push Limit in front of ForEach with flatten. It is based on the assumption that ForEach with a flatten always increase the number of records. However, this is a false assumption. In the case that there is empty bags inside input tuple, the number of output records can be 0, thus less than input records.
> We have no way to know whether there is an empty bag in the input at optimization time. So the only solution is not to push Limit in front of ForEach with flatten
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-362) Limit can not push in front of ForEach
with flatten
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-362:
-------------------------------
Status: Patch Available (was: Open)
> Limit can not push in front of ForEach with flatten
> ---------------------------------------------------
>
> Key: PIG-362
> URL: https://issues.apache.org/jira/browse/PIG-362
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: types_branch
>
> Attachments: PIG-362-2.patch, PIG-362.patch
>
>
> Currently logical optimizer will push Limit in front of ForEach with flatten. It is based on the assumption that ForEach with a flatten always increase the number of records. However, this is a false assumption. In the case that there is empty bags inside input tuple, the number of output records can be 0, thus less than input records.
> We have no way to know whether there is an empty bag in the input at optimization time. So the only solution is not to push Limit in front of ForEach with flatten
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-362) Limit can not push in front of ForEach
with flatten
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-362:
---------------------------
Attachment: PIG-362.patch
> Limit can not push in front of ForEach with flatten
> ---------------------------------------------------
>
> Key: PIG-362
> URL: https://issues.apache.org/jira/browse/PIG-362
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: types_branch
>
> Attachments: PIG-362.patch
>
>
> Currently logical optimizer will push Limit in front of ForEach with flatten. It is based on the assumption that ForEach with a flatten always increase the number of records. However, this is a false assumption. In the case that there is empty bags inside input tuple, the number of output records can be 0, thus less than input records.
> We have no way to know whether there is an empty bag in the input at optimization time. So the only solution is not to push Limit in front of ForEach with flatten
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-362) Limit can not push in front of ForEach
with flatten
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622918#action_12622918 ]
Olga Natkovich commented on PIG-362:
------------------------------------
Daniel, could you add a unit test for this to your patch. thanks
> Limit can not push in front of ForEach with flatten
> ---------------------------------------------------
>
> Key: PIG-362
> URL: https://issues.apache.org/jira/browse/PIG-362
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: types_branch
>
> Attachments: PIG-362.patch
>
>
> Currently logical optimizer will push Limit in front of ForEach with flatten. It is based on the assumption that ForEach with a flatten always increase the number of records. However, this is a false assumption. In the case that there is empty bags inside input tuple, the number of output records can be 0, thus less than input records.
> We have no way to know whether there is an empty bag in the input at optimization time. So the only solution is not to push Limit in front of ForEach with flatten
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-362) Limit can not push in front of ForEach
with flatten
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-362:
---------------------------
Attachment: PIG-362-2.patch
Include test case
> Limit can not push in front of ForEach with flatten
> ---------------------------------------------------
>
> Key: PIG-362
> URL: https://issues.apache.org/jira/browse/PIG-362
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: types_branch
>
> Attachments: PIG-362-2.patch, PIG-362.patch
>
>
> Currently logical optimizer will push Limit in front of ForEach with flatten. It is based on the assumption that ForEach with a flatten always increase the number of records. However, this is a false assumption. In the case that there is empty bags inside input tuple, the number of output records can be 0, thus less than input records.
> We have no way to know whether there is an empty bag in the input at optimization time. So the only solution is not to push Limit in front of ForEach with flatten
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.