You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Ashutosh Chauhan (JIRA)" <ji...@apache.org> on 2009/06/19 20:22:07 UTC
[jira] Created: (PIG-858) Order By followed by "replicated" join
fails while compiling MR-plan from physical plan
Order By followed by "replicated" join fails while compiling MR-plan from physical plan
---------------------------------------------------------------------------------------
Key: PIG-858
URL: https://issues.apache.org/jira/browse/PIG-858
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.3.0
Reporter: Ashutosh Chauhan
Fix For: 0.4.0
Consider the query:
{code}
A = load 'a';
B = order A by $0;
C = join A by $0, B by $0;
explain C;
{code}
works. But if replicated join is used instead
{code}
A = load 'a';
B = order A by $0;
C = join A by $0, B by $0 using "replicated";
explain C;
{code}
this fails with ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2034: Error compiling operator POFRJoin
relevant stacktrace:
{code}
Caused by: java.lang.RuntimeException: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:306)
at org.apache.pig.PigServer.explain(PigServer.java:574)
... 8 more
Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:942)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:173)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:342)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:327)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:233)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:301)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.explain(MapReduceLauncher.java:278)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:303)
... 9 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:901)
... 16 more
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-858) Order By followed by "replicated" join
fails while compiling MR-plan from physical plan
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765601#action_12765601 ]
Alan Gates commented on PIG-858:
--------------------------------
Mostly looks straight forward and passes all the tests. You made a number of changes in MRCompiler.visitUnion. I don't understand what exactly you were changing there. Could you give a brief overview of those changes?
> Order By followed by "replicated" join fails while compiling MR-plan from physical plan
> ---------------------------------------------------------------------------------------
>
> Key: PIG-858
> URL: https://issues.apache.org/jira/browse/PIG-858
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Fix For: 0.6.0
>
> Attachments: pig-858.patch
>
>
> Consider the query:
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0;
> explain C;
> {code}
> works. But if replicated join is used instead
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0 using "replicated";
> explain C;
> {code}
> this fails with ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2034: Error compiling operator POFRJoin
> relevant stacktrace:
> {code}
> Caused by: java.lang.RuntimeException: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:306)
> at org.apache.pig.PigServer.explain(PigServer.java:574)
> ... 8 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:942)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:173)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:342)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:327)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:233)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:301)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.explain(MapReduceLauncher.java:278)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:303)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:901)
> ... 16 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-858) Order By followed by "replicated" join
fails while compiling MR-plan from physical plan
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765735#action_12765735 ]
Ashutosh Chauhan commented on PIG-858:
--------------------------------------
Its been a while since I did that patch. So, bit more clarification: We are interested in finding PO which corresponds to "fragment" PO input of POFRJoin. This PO is already compiled and is in one the MROper. Earlier we will iterate through compiledInputs array trying to match this PO with PO contained in each MROperator. This fails as discussed in previous comments. With this change, since we keep track of MR operator with each physical operator it need not to do that but can simply look up for MROper corresponding to "fragment" PO in the phyToMROpMap.
> Order By followed by "replicated" join fails while compiling MR-plan from physical plan
> ---------------------------------------------------------------------------------------
>
> Key: PIG-858
> URL: https://issues.apache.org/jira/browse/PIG-858
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Fix For: 0.6.0
>
> Attachments: pig-858.patch
>
>
> Consider the query:
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0;
> explain C;
> {code}
> works. But if replicated join is used instead
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0 using "replicated";
> explain C;
> {code}
> this fails with ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2034: Error compiling operator POFRJoin
> relevant stacktrace:
> {code}
> Caused by: java.lang.RuntimeException: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:306)
> at org.apache.pig.PigServer.explain(PigServer.java:574)
> ... 8 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:942)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:173)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:342)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:327)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:233)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:301)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.explain(MapReduceLauncher.java:278)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:303)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:901)
> ... 16 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-858) Order By followed by "replicated" join
fails while compiling MR-plan from physical plan
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-858:
---------------------------------
Fix Version/s: 0.6.0
Affects Version/s: (was: 0.3.0)
0.4.0
Status: Patch Available (was: Open)
> Order By followed by "replicated" join fails while compiling MR-plan from physical plan
> ---------------------------------------------------------------------------------------
>
> Key: PIG-858
> URL: https://issues.apache.org/jira/browse/PIG-858
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Fix For: 0.6.0
>
> Attachments: pig-858.patch
>
>
> Consider the query:
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0;
> explain C;
> {code}
> works. But if replicated join is used instead
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0 using "replicated";
> explain C;
> {code}
> this fails with ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2034: Error compiling operator POFRJoin
> relevant stacktrace:
> {code}
> Caused by: java.lang.RuntimeException: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:306)
> at org.apache.pig.PigServer.explain(PigServer.java:574)
> ... 8 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:942)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:173)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:342)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:327)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:233)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:301)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.explain(MapReduceLauncher.java:278)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:303)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:901)
> ... 16 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-858) Order By followed by "replicated" join
fails while compiling MR-plan from physical plan
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan reassigned PIG-858:
------------------------------------
Assignee: Ashutosh Chauhan
> Order By followed by "replicated" join fails while compiling MR-plan from physical plan
> ---------------------------------------------------------------------------------------
>
> Key: PIG-858
> URL: https://issues.apache.org/jira/browse/PIG-858
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.3.0
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Attachments: pig-858.patch
>
>
> Consider the query:
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0;
> explain C;
> {code}
> works. But if replicated join is used instead
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0 using "replicated";
> explain C;
> {code}
> this fails with ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2034: Error compiling operator POFRJoin
> relevant stacktrace:
> {code}
> Caused by: java.lang.RuntimeException: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:306)
> at org.apache.pig.PigServer.explain(PigServer.java:574)
> ... 8 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:942)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:173)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:342)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:327)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:233)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:301)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.explain(MapReduceLauncher.java:278)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:303)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:901)
> ... 16 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-858) Order By followed by "replicated" join
fails while compiling MR-plan from physical plan
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760141#action_12760141 ]
Hadoop QA commented on PIG-858:
-------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419592/pig-858.patch
against trunk revision 819057.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/47/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/47/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/47/console
This message is automatically generated.
> Order By followed by "replicated" join fails while compiling MR-plan from physical plan
> ---------------------------------------------------------------------------------------
>
> Key: PIG-858
> URL: https://issues.apache.org/jira/browse/PIG-858
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Fix For: 0.6.0
>
> Attachments: pig-858.patch
>
>
> Consider the query:
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0;
> explain C;
> {code}
> works. But if replicated join is used instead
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0 using "replicated";
> explain C;
> {code}
> this fails with ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2034: Error compiling operator POFRJoin
> relevant stacktrace:
> {code}
> Caused by: java.lang.RuntimeException: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:306)
> at org.apache.pig.PigServer.explain(PigServer.java:574)
> ... 8 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:942)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:173)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:342)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:327)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:233)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:301)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.explain(MapReduceLauncher.java:278)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:303)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:901)
> ... 16 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-858) Order By followed by "replicated" join
fails while compiling MR-plan from physical plan
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765334#action_12765334 ]
Alan Gates commented on PIG-858:
--------------------------------
I'm reviewing this patch.
> Order By followed by "replicated" join fails while compiling MR-plan from physical plan
> ---------------------------------------------------------------------------------------
>
> Key: PIG-858
> URL: https://issues.apache.org/jira/browse/PIG-858
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Fix For: 0.6.0
>
> Attachments: pig-858.patch
>
>
> Consider the query:
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0;
> explain C;
> {code}
> works. But if replicated join is used instead
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0 using "replicated";
> explain C;
> {code}
> this fails with ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2034: Error compiling operator POFRJoin
> relevant stacktrace:
> {code}
> Caused by: java.lang.RuntimeException: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:306)
> at org.apache.pig.PigServer.explain(PigServer.java:574)
> ... 8 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:942)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:173)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:342)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:327)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:233)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:301)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.explain(MapReduceLauncher.java:278)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:303)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:901)
> ... 16 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-858) Order By followed by "replicated" join
fails while compiling MR-plan from physical plan
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated PIG-858:
---------------------------------
Attachment: pig-858.patch
Patch as discussed in previous comment. Also included are test cases, where blocking operator (order-by, distinct) occurs before FRjoin.
> Order By followed by "replicated" join fails while compiling MR-plan from physical plan
> ---------------------------------------------------------------------------------------
>
> Key: PIG-858
> URL: https://issues.apache.org/jira/browse/PIG-858
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.3.0
> Reporter: Ashutosh Chauhan
> Attachments: pig-858.patch
>
>
> Consider the query:
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0;
> explain C;
> {code}
> works. But if replicated join is used instead
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0 using "replicated";
> explain C;
> {code}
> this fails with ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2034: Error compiling operator POFRJoin
> relevant stacktrace:
> {code}
> Caused by: java.lang.RuntimeException: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:306)
> at org.apache.pig.PigServer.explain(PigServer.java:574)
> ... 8 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:942)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:173)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:342)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:327)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:233)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:301)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.explain(MapReduceLauncher.java:278)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:303)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:901)
> ... 16 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-858) Order By followed by "replicated" join
fails while compiling MR-plan from physical plan
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-858:
---------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Fix checked in. Thanks Ashutosh for the patch and explanation.
> Order By followed by "replicated" join fails while compiling MR-plan from physical plan
> ---------------------------------------------------------------------------------------
>
> Key: PIG-858
> URL: https://issues.apache.org/jira/browse/PIG-858
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Fix For: 0.6.0
>
> Attachments: pig-858.patch
>
>
> Consider the query:
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0;
> explain C;
> {code}
> works. But if replicated join is used instead
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0 using "replicated";
> explain C;
> {code}
> this fails with ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2034: Error compiling operator POFRJoin
> relevant stacktrace:
> {code}
> Caused by: java.lang.RuntimeException: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:306)
> at org.apache.pig.PigServer.explain(PigServer.java:574)
> ... 8 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:942)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:173)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:342)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:327)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:233)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:301)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.explain(MapReduceLauncher.java:278)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:303)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:901)
> ... 16 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-858) Order By followed by "replicated" join
fails while compiling MR-plan from physical plan
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733057#action_12733057 ]
Ashutosh Chauhan commented on PIG-858:
--------------------------------------
While POFRJoin is getting compiled in MRCompiler, it needs to identify for each of its
predecessor in physical plan of which compiled MROperator they are part of. Currently, it is
assumed to be one of the compiledInputs(an array of MRoper which are immediate predecessor of current MROper in MROper DAG).
Mostly this is true, but in cases where one physical operator results in two or more MR operator, this may not be true, as is the
case here. When there is an order-by before FRJoin; one of the inputs of POFRJoin will be
POSort, but POSort operator will be in the first MROper of the two generated MROperator
and thus will not be found in compiledInputs (which contains second MROper). Thus,
current way of identifying corresponding MRoper of a physical operator is unreliable.
This bug also affects the implementation of merge-sort join
https://issues.apache.org/jira/browse/PIG-845 . Since POMergeJoin needs to know which MROper
corresponds to its left input and which one corresponds to its right. It can do so by looking
into compiledInputs as long as there is no order-by (or similiar PO which results in
multiple MROper) as its predecessors. Doing order-by before using merge
join is however a natural use-case there.
Proposal is to introduce a new private member variable in MRCompiler phyToMROperMap
(similiar to logToPhyMap) using which leaf MROper for a given
physical operator can be identified. Thoughts?
> Order By followed by "replicated" join fails while compiling MR-plan from physical plan
> ---------------------------------------------------------------------------------------
>
> Key: PIG-858
> URL: https://issues.apache.org/jira/browse/PIG-858
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.3.0
> Reporter: Ashutosh Chauhan
> Fix For: 0.4.0
>
>
> Consider the query:
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0;
> explain C;
> {code}
> works. But if replicated join is used instead
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0 using "replicated";
> explain C;
> {code}
> this fails with ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2034: Error compiling operator POFRJoin
> relevant stacktrace:
> {code}
> Caused by: java.lang.RuntimeException: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:306)
> at org.apache.pig.PigServer.explain(PigServer.java:574)
> ... 8 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:942)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:173)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:342)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:327)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:233)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:301)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.explain(MapReduceLauncher.java:278)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:303)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:901)
> ... 16 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-858) Order By followed by "replicated" join
fails while compiling MR-plan from physical plan
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765720#action_12765720 ]
Ashutosh Chauhan commented on PIG-858:
--------------------------------------
visitUnion has same changes as others visit functions, that is it adds MR Operator corresponding to POUnion in phyToMROpMap map. Real changes are in visitFRJoin. Earlier in visitFRJoin, it used to look in compiledInputs array of MROper one by one trying to match MROPer leaf PO with POFRJoin using operator key. Now, it doesn't need to do that it can simply lookup in the phyToMROpMap.
> Order By followed by "replicated" join fails while compiling MR-plan from physical plan
> ---------------------------------------------------------------------------------------
>
> Key: PIG-858
> URL: https://issues.apache.org/jira/browse/PIG-858
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.4.0
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Fix For: 0.6.0
>
> Attachments: pig-858.patch
>
>
> Consider the query:
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0;
> explain C;
> {code}
> works. But if replicated join is used instead
> {code}
> A = load 'a';
> B = order A by $0;
> C = join A by $0, B by $0 using "replicated";
> explain C;
> {code}
> this fails with ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2034: Error compiling operator POFRJoin
> relevant stacktrace:
> {code}
> Caused by: java.lang.RuntimeException: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:306)
> at org.apache.pig.PigServer.explain(PigServer.java:574)
> ... 8 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2034: Error compiling operator POFRJoin
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:942)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:173)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:342)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:327)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:233)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:301)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.explain(MapReduceLauncher.java:278)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.explain(HExecutionEngine.java:303)
> ... 9 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:901)
> ... 16 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.