You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Koji Noguchi (JIRA)" <ji...@apache.org> on 2012/10/12 21:03:03 UTC
[jira] [Created] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Koji Noguchi created PIG-2968:
---------------------------------
Summary: ColumnMapKeyPrune fails to prune a subtree inside foreach
Key: PIG-2968
URL: https://issues.apache.org/jira/browse/PIG-2968
Project: Pig
Issue Type: Bug
Reporter: Koji Noguchi
Assignee: Koji Noguchi
Priority: Minor
Sample code
{noformat}
$ cat test/foreach.pig
daily = load 'nyse' as (exchange, symbol);
grpd = group daily by exchange;
uniquecnt = foreach grpd {
sym = daily.symbol;
uniq_sym = distinct sym;
generate group, uniq_sym;
};
another = FOREACH uniquecnt GENERATE group;
explain another;
{noformat}
This breaks when it tries to prune uniq_sym->sym->innerload_daily
bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-2968:
------------------------------
Attachment: pig-2968-trunk_v01.txt
ConcurrentModificationException is happening in ColumnPruneVisitor.removeSubTree
{noformat}
List<Operator> ll = p.getPredecessors(op);
if (ll != null) {
for(Operator pred: ll) {
removeSubTree((LogicalRelationalOperator)pred);
}
}
{noformat}
where List ll is updated within (recursive-)removeSubTree
while being traversed.
Attaching a patch that creates a shallow copy of the list.
For the test, attaching one that will fail with Exception. But I couldn't come up with _expected_ raw query that matches the pruned result.
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Attachments: pig-2968-trunk_v01.txt
>
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486365#comment-13486365 ]
Cheolsoo Park commented on PIG-2968:
------------------------------------
Hi Rohini, sure I can wait until your concern is addressed.
If Koji can improve the test case, that will be of course great. But since I don't have a better suggestion, I won't insist.
Thanks!
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Components: parser
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.11
>
> Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt
>
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Rohini Palaniswamy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486932#comment-13486932 ]
Rohini Palaniswamy commented on PIG-2968:
-----------------------------------------
bq. I tried simply taking out the lines but nested foreach with pruned inputs were different from a simple foreach with just one output. (one extra foreach on former)
Got it Koji. ColumnPruneVisitor.addForEachIfNecessary adds an additional foreach to prune columns for optimization even though the next statement from user does exactly the same. Don't think it is possible to have a equivalent query as the attached LOForEach is without an alias. I am good Cheolsoo. You can go ahead with the commit.
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Components: parser
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.11
>
> Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt, pig-2968-trunk_v03.txt
>
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Rohini Palaniswamy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486344#comment-13486344 ]
Rohini Palaniswamy commented on PIG-2968:
-----------------------------------------
Koji/Cheolsoo,
bq. But I couldn't come up with expected raw query that matches the pruned result.
I missed commenting on this in the initial review. Should we take a crack at this before committing this. Or at least some Assert?
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Components: parser
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.11
>
> Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt
>
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Rohini Palaniswamy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475950#comment-13475950 ]
Rohini Palaniswamy commented on PIG-2968:
-----------------------------------------
ll.toArray(new Operator[ll.size()]) can be done instead of ll.toArray(new Operator[0]) for little more efficiency. Looks good otherwise.
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Components: parser
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Attachments: pig-2968-trunk_v01.txt
>
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-2968:
------------------------------
Component/s: parser
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Components: parser
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Attachments: pig-2968-trunk_v01.txt
>
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-2968:
------------------------------
Attachment: pig-2968-trunk_v03.txt
bq. Actually, can you please fix the indentation of testPruneSubTreeForEach() using 4 spaces?
Changed. (Sorry, I recently learned it's 4 spaces instead of 2.)
bq. Or at least some Assert?
Added a catch and fail call.
bq. Should we take a crack at this before committing this
I tried simply taking out the lines but nested foreach with pruned inputs were different from a simple foreach with just one output. (one extra foreach on former)
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Components: parser
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.11
>
> Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt, pig-2968-trunk_v03.txt
>
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486342#comment-13486342 ]
Cheolsoo Park commented on PIG-2968:
------------------------------------
Actually, can you please fix the indentation of testPruneSubTreeForEach() using 4 spaces? We're trying to clean up white spaces in code base.
Thanks!
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Components: parser
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.11
>
> Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt
>
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-2968:
------------------------------
Status: Patch Available (was: Open)
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Components: parser
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt
>
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486329#comment-13486329 ]
Cheolsoo Park commented on PIG-2968:
------------------------------------
+1.
I will commit it after running tests. Thanks Koji!
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Components: parser
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.11
>
> Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt
>
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-2968:
------------------------------
Attachment: pig-2968-trunk_v02.txt
bq. ll.toArray(new Operator[ll.size()]) can be done instead of ll.toArray(new Operator[0])
Uploading with the above change.
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Components: parser
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt
>
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475247#comment-13475247 ]
Koji Noguchi commented on PIG-2968:
-----------------------------------
Log showing
{noformat}
$ cat /Users/knoguchi/git/pig/pig_1350068049281.logPig Stack Trace
---------------
ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1067: Unable to explain alias another at org.apache.pig.PigServer.explain(PigServer.java:999)
at org.apache.pig.tools.grunt.GruntParser.explainCurrentBatch(GruntParser.java:398)
at org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:330)
at org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:293)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.Explain(PigScriptParser.java:715) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:342)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84) at org.apache.pig.Main.run(Main.java:604)
at org.apache.pig.Main.main(Main.java:154)
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:277)
at org.apache.pig.PigServer.compilePp(PigServer.java:1322) at org.apache.pig.PigServer.explain(PigServer.java:984)
... 10 more
Caused by: java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at org.apache.pig.newplan.logical.rules.ColumnPruneVisitor.removeSubTree(ColumnPruneVisitor.java:451)
at org.apache.pig.newplan.logical.rules.ColumnPruneVisitor.removeSubTree(ColumnPruneVisitor.java:452)
at org.apache.pig.newplan.logical.rules.ColumnPruneVisitor.visit(ColumnPruneVisitor.java:431)
at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:76)
at org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.newplan.logical.rules.ColumnMapKeyPrune$ColumnMapKeyPruneTransformer.transform(ColumnMapKeyPrune.java:141)
at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
... 13 more
================================================================================
{noformat}
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-2968:
------------------------------
Fix Version/s: 0.11
I'd like to see this fixed in 0.11.
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Components: parser
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.11
>
> Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt
>
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2968) ColumnMapKeyPrune fails to prune a
subtree inside foreach
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2968:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to 0.11/trunk. Thanks Koji!
> ColumnMapKeyPrune fails to prune a subtree inside foreach
> ---------------------------------------------------------
>
> Key: PIG-2968
> URL: https://issues.apache.org/jira/browse/PIG-2968
> Project: Pig
> Issue Type: Bug
> Components: parser
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.11
>
> Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt, pig-2968-trunk_v03.txt
>
>
> Sample code
> {noformat}
> $ cat test/foreach.pig
> daily = load 'nyse' as (exchange, symbol);
> grpd = group daily by exchange;
> uniquecnt = foreach grpd {
> sym = daily.symbol;
> uniq_sym = distinct sym;
> generate group, uniq_sym;
> };
> another = FOREACH uniquecnt GENERATE group;
> explain another;
> {noformat}
> This breaks when it tries to prune uniq_sym->sym->innerload_daily
> bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira