You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2009/12/08 03:11:18 UTC
[jira] Created: (PIG-1132) Column Pruner issues in dealing with
unprunable loader
Column Pruner issues in dealing with unprunable loader
------------------------------------------------------
Key: PIG-1132
URL: https://issues.apache.org/jira/browse/PIG-1132
Project: Pig
Issue Type: Bug
Affects Versions: 0.4.0
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: 0.6.0
The following script fail:
{code}
a = load '1.txt' using BinStorage() as (a0, a1, a2);
b = foreach a generate a2, a0, a1;
c = foreach b generate a0, a2;
explain c;
{code}
Error message:
ERROR 2163: Error during fixing projections. Could not locate replacement column for column: 1 in the new predecessor at org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
at org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
... 19 more
Also even we fix the exception, the result is wrong.
Thanks Thejas for finding a reproducible test case for PigLatin
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1132) Column Pruner issues in dealing with
unprunable loader
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1132:
----------------------------
Attachment: PIG-1132-1.patch
> Column Pruner issues in dealing with unprunable loader
> ------------------------------------------------------
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement column for column: 1 in the new predecessor at org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1132) Column Pruner issues in dealing with
unprunable loader
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787701#action_12787701 ]
Alan Gates commented on PIG-1132:
---------------------------------
Is this bug specific to a script with two foreachs in a row, or will it manifest with any script with more than one foreach? So would:
A = load
B = foreach ..
C = filter
D = group
E = foreach
trigger this bug as well?
> Column Pruner issues in dealing with unprunable loader
> ------------------------------------------------------
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement column for column: 1 in the new predecessor at org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1132) Column Pruner issues in dealing with
unprunable loader
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787339#action_12787339 ]
Hadoop QA commented on PIG-1132:
--------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427278/PIG-1132-1.patch
against trunk revision 888186.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/102/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/102/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/102/console
This message is automatically generated.
> Column Pruner issues in dealing with unprunable loader
> ------------------------------------------------------
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement column for column: 1 in the new predecessor at org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1132) Column Pruner issues in dealing with
unprunable loader
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787770#action_12787770 ]
Alan Gates commented on PIG-1132:
---------------------------------
Thanks, that clears up several questions I had.
+1, patch looks good.
> Column Pruner issues in dealing with unprunable loader
> ------------------------------------------------------
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement column for column: 1 in the new predecessor at org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1132) Column Pruner issues in dealing with
unprunable loader
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1132:
----------------------------
Attachment: PIG-1132-2.patch
Ignore the first patch.
> Column Pruner issues in dealing with unprunable loader
> ------------------------------------------------------
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement column for column: 1 in the new predecessor at org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1132) Column Pruner issues in dealing with
unprunable loader
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1132:
----------------------------
Status: Patch Available (was: Open)
> Column Pruner issues in dealing with unprunable loader
> ------------------------------------------------------
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement column for column: 1 in the new predecessor at org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1132) Column Pruner issues in dealing with
unprunable loader
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787506#action_12787506 ]
Hadoop QA commented on PIG-1132:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427303/PIG-1132-2.patch
against trunk revision 888186.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/104/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/104/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/104/console
This message is automatically generated.
> Column Pruner issues in dealing with unprunable loader
> ------------------------------------------------------
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement column for column: 1 in the new predecessor at org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1132) Column Pruner issues in dealing with
unprunable loader
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1132:
----------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Core test failures are due to port conflict. Test manually and it is successful. Patch committed to both trunk and 0.6 branch. Thanks Alan!
> Column Pruner issues in dealing with unprunable loader
> ------------------------------------------------------
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement column for column: 1 in the new predecessor at org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1132) Column Pruner issues in dealing with
unprunable loader
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1132:
----------------------------
Status: Patch Available (was: Open)
> Column Pruner issues in dealing with unprunable loader
> ------------------------------------------------------
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement column for column: 1 in the new predecessor at org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1132) Column Pruner issues in dealing with
unprunable loader
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1132:
----------------------------
Status: Open (was: Patch Available)
> Column Pruner issues in dealing with unprunable loader
> ------------------------------------------------------
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement column for column: 1 in the new predecessor at org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1132) Column Pruner issues in dealing with
unprunable loader
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787716#action_12787716 ]
Daniel Dai commented on PIG-1132:
---------------------------------
Yes, it could happen for any logical operators after Load. The key issue for this problem is OperatorPlan.insertBetween assume logical plan in a consistent state. What I want to do here is inserting a LOForEach after LOLoad if the loader do not support pruning. I will need to achieve this in two steps:
1. insert a LOForEach after LoLoad
2. prune columns the logical operator below
I have to leave the logical plan in an inconsistent state in the middle. If I insert LOForEach first, then the underlying logical operator has not been pruned yet; If I prune underlying logical operator first, then there is a schema gap between underlying logical operator and LOLoad. Both situations are not allowed for insertBetween. I have to use a trick to get around this problem.
> Column Pruner issues in dealing with unprunable loader
> ------------------------------------------------------
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement column for column: 1 in the new predecessor at org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.