You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Vivek Padmanabhan (Created) (JIRA)" <ji...@apache.org> on 2011/09/30 14:41:45 UTC
[jira] [Created] (PIG-2312) NPE when relation and column share the
same name and used in Nested Foreach
NPE when relation and column share the same name and used in Nested Foreach
----------------------------------------------------------------------------
Key: PIG-2312
URL: https://issues.apache.org/jira/browse/PIG-2312
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Vivek Padmanabhan
With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
while compiling.
The below is the trace;
{code}
java.lang.NullPointerException
at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
at org.apache.pig.PigServer.execute(PigServer.java:1293)
at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:553)
at org.apache.pig.Main.main(Main.java:108)
{code}
This could be reproduced with the below script
{code}
f3 = load 'input.txt' as (a1:chararray);
A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
B = GROUP A BY f1;
C = FOREACH B {
zip_ordered = ORDER A BY f3 ASC;
GENERATE
FLATTEN(group) AS f1,
A.(f3, ct),
COUNT(zip_ordered),
SUM(A.ct) AS total;
};
STORE C INTO 'deletemeanytimeplease';
{code}
Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2312) NPE when relation and column share the
same name and used in Nested Foreach
Posted by "Vivek Padmanabhan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vivek Padmanabhan updated PIG-2312:
-----------------------------------
Status: Open (was: Patch Available)
Yes, I missed this basic case. Thanks for pointing out. I will cancel this patch.
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2312) NPE when relation and column share
the same name and used in Nested Foreach
Posted by "Vivek Padmanabhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143058#comment-13143058 ]
Vivek Padmanabhan commented on PIG-2312:
----------------------------------------
Hi Olga,Sorry for the delayed response. I am attempting another patch for this issue.
I have added some more test cases to cover the scenarios that I have faced while debugging.
The patch includes changes in LogicalPlanGenerator.g and LogicalPlanBuilder (added a new map of operators).
Apart from the general case mentioned in the above comment, I have tried to make the patch work with queries like;
a)
f3 = load ....
A = load 'input' using PigStorage(',') as (f1:chararray, f2:chararray, f3:chararray,ct:long);
B = GROUP A BY f1;
C = FOREACH B { dis = DISTINCT A ;
ordered = ORDER dis BY f3 ASC; .....
b)
f3 = load ....
A = load 'input' using PigStorage(',') as (f1:chararray, f2:chararray, f3:chararray,ct:long);
B = GROUP A BY f1;
C = FOREACH B { f3 = DISTINCT A ;
ordered = ORDER f3 BY f3 ASC; .....
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch, PIG-2312_2.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2312) NPE when relation and column share the
same name and used in Nested Foreach
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2312:
----------------------------
Fix Version/s: (was: 0.10)
0.11
The fix is quite involved and we need more time to figure the best way. Unlink from 0.10.
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.11
>
> Attachments: PIG-2312_1.patch, PIG-2312_2.patch, PIG-2312_3.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2312) NPE when relation and column share the
same name and used in Nested Foreach
Posted by "Vivek Padmanabhan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vivek Padmanabhan updated PIG-2312:
-----------------------------------
Status: Patch Available (was: Open)
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch, PIG-2312_2.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2312) NPE when relation and column share the
same name and used in Nested Foreach
Posted by "Vivek Padmanabhan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vivek Padmanabhan updated PIG-2312:
-----------------------------------
Attachment: PIG-2312_3.patch
Attaching another patch to fix the failures. I am not sure whether I have complicated the code,but had to write a logic to find the actual input operator.
(Ran test-core with this patch except for
TestEvalPipeline2 and TestMergeJoin which are timing out in my env)
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch, PIG-2312_2.patch, PIG-2312_3.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2312) NPE when relation and column share
the same name and used in Nested Foreach
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145149#comment-13145149 ]
Daniel Dai commented on PIG-2312:
---------------------------------
This should be the right fix. It would be better to add a test case for cogroup as well.
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch, PIG-2312_2.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2312) NPE when relation and column share
the same name and used in Nested Foreach
Posted by "Gianmarco De Francisci Morales (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146271#comment-13146271 ]
Gianmarco De Francisci Morales commented on PIG-2312:
-----------------------------------------------------
Strange, the syntax of this last snippet should not be allowed as sc_load.$0 is a projection and not a scalar. Do you get an NPE also with this?
Regarding the approach in the patch, I am not sure we want to disable the scalars in nested foreach. One of the use cases is trimming an inner bag to a fixed size for all the tuples in the outer bag.
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch, PIG-2312_2.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2312) NPE when relation and column share the
same name and used in Nested Foreach
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Le Dem updated PIG-2312:
-------------------------------
Fix Version/s: (was: 0.11)
Release Note: Unlinking from 0.11 for the same reason
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Attachments: PIG-2312_1.patch, PIG-2312_2.patch, PIG-2312_3.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2312) NPE when relation and column share the
same name and used in Nested Foreach
Posted by "Vivek Padmanabhan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vivek Padmanabhan updated PIG-2312:
-----------------------------------
Attachment: PIG-2312_1.patch
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2312) NPE when relation and column share
the same name and used in Nested Foreach
Posted by "Vivek Padmanabhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147168#comment-13147168 ]
Vivek Padmanabhan commented on PIG-2312:
----------------------------------------
The second case is failing with the patch attached. And it should not be disabling scalar in nested foreach. I am doing a rework on the patch.
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch, PIG-2312_2.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2312) NPE when relation and column share the
same name and used in Nested Foreach
Posted by "Vivek Padmanabhan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vivek Padmanabhan updated PIG-2312:
-----------------------------------
Assignee: Vivek Padmanabhan
Status: Patch Available (was: Open)
Attempting for a patch for this issue.
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2312) NPE when relation and column share
the same name and used in Nested Foreach
Posted by "Thejas M Nair (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125551#comment-13125551 ]
Thejas M Nair commented on PIG-2312:
------------------------------------
The patch PIG-2312_1.patch disables use of scalar within operators inside a nested-foreach.
For example, for the following query -
{code}
sc_load = load 'x' as (x, y , z);
l = load 'x' as (x,y , A : bag {t : ( i : int)} );
f = foreach l {
fil = filter A by sc_load.$0 > $0 ;
generate fil;
}
{code}
it throws the following exception -
{code}
<line 5, column 20> Invalid field projection. Projected field [sc_load] does not exist in schema: i:int.
{code}
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2312) NPE when relation and column share
the same name and used in Nested Foreach
Posted by "Thejas M Nair (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125552#comment-13125552 ]
Thejas M Nair commented on PIG-2312:
------------------------------------
The reason for the original bug is that the logic that determines if an alias used is to be mapped to a relation is not checking the input schema for the nested operation. It is looking for that alias in the schema of input relation of the foreach, instead of looking for the schema in the input relation for the order-by .
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2312) NPE when relation and column share
the same name and used in Nested Foreach
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145176#comment-13145176 ]
Daniel Dai commented on PIG-2312:
---------------------------------
TestLimitVariable.testNestedLimitVariable1 fail with the patch. Can you take a look?
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch, PIG-2312_2.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2312) NPE when relation and column share the
same name and used in Nested Foreach
Posted by "Vivek Padmanabhan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vivek Padmanabhan updated PIG-2312:
-----------------------------------
Attachment: PIG-2312_2.patch
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch, PIG-2312_2.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2312) NPE when relation and column share
the same name and used in Nested Foreach
Posted by "Olga Natkovich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142674#comment-13142674 ]
Olga Natkovich commented on PIG-2312:
-------------------------------------
Hi Vivek,
Are you planning to re-submit the patch?
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2312) NPE when relation and column share
the same name and used in Nested Foreach
Posted by "Olga Natkovich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122423#comment-13122423 ]
Olga Natkovich commented on PIG-2312:
-------------------------------------
Did this used to work in 0.8?
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Fix For: 0.10
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2312) NPE when relation and column share
the same name and used in Nested Foreach
Posted by "Vivek Padmanabhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122508#comment-13122508 ]
Vivek Padmanabhan commented on PIG-2312:
----------------------------------------
Oops, extremely sorry, I thought I had mentioned it in my previous comment.Yes, this works fine in 0.8
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Fix For: 0.10
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2312) NPE when relation and column share
the same name and used in Nested Foreach
Posted by "Vivek Padmanabhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120848#comment-13120848 ]
Vivek Padmanabhan commented on PIG-2312:
----------------------------------------
I think the logical plan generator is creating a ScalarExpression instead of ProjectExpression which causes this NPE.
LogicalPlanGenerator.g
Operator op = builder.lookupOperator( alias );
if( op != null && ( schema == null || schema.getFieldPosition( alias ) == -1 ) ) {
$expr = new ScalarExpression( plan, op,
inForeachPlan ? $foreach_clause::foreachOp : $GScope::currentOp );
$expr.setLocation( loc );
}
The lookupOperator returns the f3 LOAD operator and schema.getFieldPosition( alias ) return -1 since it searches only the outer fields (group and A{} in this case )
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Fix For: 0.10
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2312) NPE when relation and column share
the same name and used in Nested Foreach
Posted by "Vivek Padmanabhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146240#comment-13146240 ]
Vivek Padmanabhan commented on PIG-2312:
----------------------------------------
The patch is colliding with the new scalar support for limit in nested foreach. Moreover, I found another case which is failing ;
{code}
sc_load = load 'input1' as (x, y , z);
l = load 'input2' as (x,y , A : bag {t : ( i : int)} );
f = foreach l {
lim = limit A sc_load.$0;
fil = filter lim by sc_load.$0 > $0 ; generate fil; }
{code}
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2312_1.patch, PIG-2312_2.patch
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2312) NPE when relation and column share the
same name and used in Nested Foreach
Posted by "Olga Natkovich (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-2312:
--------------------------------
Fix Version/s: 0.10
> NPE when relation and column share the same name and used in Nested Foreach
> ----------------------------------------------------------------------------
>
> Key: PIG-2312
> URL: https://issues.apache.org/jira/browse/PIG-2312
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Fix For: 0.10
>
>
> With Pig0.9, if a relation and a column has the same name and if the column is used in a nested foreach, the script execution fails
> while compiling.
> The below is the trace;
> {code}
> java.lang.NullPointerException
> at org.apache.pig.newplan.logical.visitor.ScalarVisitor$1.visit(ScalarVisitor.java:63)
> at org.apache.pig.newplan.logical.expression.ScalarExpression.accept(ScalarExpression.java:109)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
> at org.apache.pig.newplan.logical.relational.LOSort.accept(LOSort.java:119)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:104)
> at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1674)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1666)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1391)
> at org.apache.pig.PigServer.execute(PigServer.java:1293)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:359)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> {code}
> This could be reproduced with the below script
> {code}
> f3 = load 'input.txt' as (a1:chararray);
> A = load '3char_1long_tab' as (f1:chararray, f2:chararray, f3:chararray,ct:long);
> B = GROUP A BY f1;
> C = FOREACH B {
> zip_ordered = ORDER A BY f3 ASC;
> GENERATE
> FLATTEN(group) AS f1,
> A.(f3, ct),
> COUNT(zip_ordered),
> SUM(A.ct) AS total;
> };
> STORE C INTO 'deletemeanytimeplease';
> {code}
> Checked with a unit test in trunk, the behavior is still same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira