You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2008/07/11 19:14:33 UTC
[jira] Created: (PIG-304) Distinct fail if previous map plan is
closed
Distinct fail if previous map plan is closed
--------------------------------------------
Key: PIG-304
URL: https://issues.apache.org/jira/browse/PIG-304
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: types_branch
Reporter: Daniel Dai
Fix For: types_branch
Attachments: distinct.patch
Distinct will fail when the previous map plan is closed. For example, the following script fail:
a = load 'a';
b = group a by $0;
c = foreach b generate $1;
d = distinct c;
dump d;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-304) Distinct fail if previous map plan is
closed
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-304:
---------------------------
Assignee: Alan Gates
Status: Patch Available (was: Open)
> Distinct fail if previous map plan is closed
> --------------------------------------------
>
> Key: PIG-304
> URL: https://issues.apache.org/jira/browse/PIG-304
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Assignee: Alan Gates
> Fix For: types_branch
>
> Attachments: distinct.patch, movelocalrearrange.patch
>
>
> Distinct will fail when the previous map plan is closed. For example, the following script fail:
> a = load 'a';
> b = group a by $0;
> c = foreach b generate $1;
> d = distinct c;
> dump d;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-304) Distinct fail if previous map plan is
closed
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-304:
---------------------------
Attachment: movelocalrearrange.patch
This patch adds a post MR compile visitor that moves local rearranges from the reducer to the next mapper. This is not the best solution. The best solution would be to make the maps somehow no-ops, so that we can avoid that stage. Ways to do that need further investigation. The next best solution would be to correct the logic in MRCompiler to place the local rearranges in the subsequent mapper instead of the reducer. But Shravan is out until Aug 4 and I don't want to change that code without his input. And seems to work.
> Distinct fail if previous map plan is closed
> --------------------------------------------
>
> Key: PIG-304
> URL: https://issues.apache.org/jira/browse/PIG-304
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Fix For: types_branch
>
> Attachments: distinct.patch, movelocalrearrange.patch
>
>
> Distinct will fail when the previous map plan is closed. For example, the following script fail:
> a = load 'a';
> b = group a by $0;
> c = foreach b generate $1;
> d = distinct c;
> dump d;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-304) Distinct fail if previous map plan is
closed
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-304:
---------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
movelocalrearrange.patch checked in.
> Distinct fail if previous map plan is closed
> --------------------------------------------
>
> Key: PIG-304
> URL: https://issues.apache.org/jira/browse/PIG-304
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Assignee: Alan Gates
> Fix For: types_branch
>
> Attachments: distinct.patch, movelocalrearrange.patch
>
>
> Distinct will fail when the previous map plan is closed. For example, the following script fail:
> a = load 'a';
> b = group a by $0;
> c = foreach b generate $1;
> d = distinct c;
> dump d;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-304) Distinct fail if previous map plan is
closed
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-304:
---------------------------
Attachment: distinct.patch
> Distinct fail if previous map plan is closed
> --------------------------------------------
>
> Key: PIG-304
> URL: https://issues.apache.org/jira/browse/PIG-304
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Fix For: types_branch
>
> Attachments: distinct.patch
>
>
> Distinct will fail when the previous map plan is closed. For example, the following script fail:
> a = load 'a';
> b = group a by $0;
> c = foreach b generate $1;
> d = distinct c;
> dump d;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-304) Distinct fail if previous map plan is
closed
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612939#action_12612939 ]
Alan Gates commented on PIG-304:
--------------------------------
The problem is more general in nature than just distinct. Queries such as the following do not work either:
a = load '/Users/gates/test/data/studenttab10' as (name, age, gpa);
b = load '/Users/gates/test/data/votertab10' as (name, age, registration, contributions);
c = filter a by age < 50;
d = filter b by age < 50;
e = cogroup c by (name, age), d by (name, age) ;
f = foreach e generate flatten(c), flatten(d);
g = group f by registration;
h = foreach g generate group, SUM(f.d::contributions);
i = order h by $1;
dump i;
The same erro is seen at the second grouping (alias g).
> Distinct fail if previous map plan is closed
> --------------------------------------------
>
> Key: PIG-304
> URL: https://issues.apache.org/jira/browse/PIG-304
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Fix For: types_branch
>
> Attachments: distinct.patch
>
>
> Distinct will fail when the previous map plan is closed. For example, the following script fail:
> a = load 'a';
> b = group a by $0;
> c = foreach b generate $1;
> d = distinct c;
> dump d;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.