You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Bill Graham (Created) (JIRA)" <ji...@apache.org> on 2012/03/06 02:51:57 UTC
[jira] [Created] (PIG-2570) LimitOptimizer fails with dynamic LIMIT
argument
LimitOptimizer fails with dynamic LIMIT argument
------------------------------------------------
Key: PIG-2570
URL: https://issues.apache.org/jira/browse/PIG-2570
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
The following script fails with the exception shown below. Passing {{-t LimitOptimizer}} makes it work, as does adding an {{ORDER}} clause before the limit.
{noformat}
A = LOAD 'data1.txt' AS (owner:chararray,pet:chararray,age:int,phone:chararray);
B = group A all;
C = foreach B generate SUM(A.age) as total;
D = foreach A generate owner, age/(double)C.total AS percentAge;
F = LIMIT D C.total/8;
DUMP F;
{noformat}
{noformat}
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule LimitOptimizer. Try -t LimitOptimizer
at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:287)
at org.apache.pig.PigServer.compilePp(PigServer.java:1317)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254)
at org.apache.pig.PigServer.storeEx(PigServer.java:953)
... 14 more
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2243: Attempt to remove operator LOLimit that is still softly connected in the plan
at org.apache.pig.newplan.BaseOperatorPlan.remove(BaseOperatorPlan.java:174)
at org.apache.pig.newplan.BaseOperatorPlan.removeAndReconnect(BaseOperatorPlan.java:449)
at org.apache.pig.newplan.logical.rules.LimitOptimizer$OptimizeLimitTransformer.transform(LimitOptimizer.java:124)
at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2570) LimitOptimizer fails with dynamic
LIMIT argument
Posted by "Bill Graham (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224603#comment-13224603 ]
Bill Graham commented on PIG-2570:
----------------------------------
+1
Thanks Daniel! This patch does the trick.
> LimitOptimizer fails with dynamic LIMIT argument
> ------------------------------------------------
>
> Key: PIG-2570
> URL: https://issues.apache.org/jira/browse/PIG-2570
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2570-1.patch, data1.txt, limit-fails.pig
>
>
> The following script fails with the exception shown below. Passing {{-t LimitOptimizer}} makes it work, as does adding an {{ORDER}} clause before the limit.
> {noformat}
> A = LOAD 'data1.txt' AS (owner:chararray,pet:chararray,age:int,phone:chararray);
> B = group A all;
> C = foreach B generate SUM(A.age) as total;
> D = foreach A generate owner, age/(double)C.total AS percentAge;
> F = LIMIT D C.total/8;
> DUMP F;
> {noformat}
> {noformat}
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule LimitOptimizer. Try -t LimitOptimizer
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:287)
> at org.apache.pig.PigServer.compilePp(PigServer.java:1317)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254)
> at org.apache.pig.PigServer.storeEx(PigServer.java:953)
> ... 14 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2243: Attempt to remove operator LOLimit that is still softly connected in the plan
> at org.apache.pig.newplan.BaseOperatorPlan.remove(BaseOperatorPlan.java:174)
> at org.apache.pig.newplan.BaseOperatorPlan.removeAndReconnect(BaseOperatorPlan.java:449)
> at org.apache.pig.newplan.logical.rules.LimitOptimizer$OptimizeLimitTransformer.transform(LimitOptimizer.java:124)
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2570) LimitOptimizer fails with dynamic
LIMIT argument
Posted by "Thejas M Nair (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228133#comment-13228133 ]
Thejas M Nair commented on PIG-2570:
------------------------------------
It is possible to have multiple scalar variables in the limit expression, this will result in limit having multiple 'soft' predecessors. This case also needs to be handled in the patch.
> LimitOptimizer fails with dynamic LIMIT argument
> ------------------------------------------------
>
> Key: PIG-2570
> URL: https://issues.apache.org/jira/browse/PIG-2570
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2570-1.patch, data1.txt, limit-fails.pig
>
>
> The following script fails with the exception shown below. Passing {{-t LimitOptimizer}} makes it work, as does adding an {{ORDER}} clause before the limit.
> {noformat}
> A = LOAD 'data1.txt' AS (owner:chararray,pet:chararray,age:int,phone:chararray);
> B = group A all;
> C = foreach B generate SUM(A.age) as total;
> D = foreach A generate owner, age/(double)C.total AS percentAge;
> F = LIMIT D C.total/8;
> DUMP F;
> {noformat}
> {noformat}
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule LimitOptimizer. Try -t LimitOptimizer
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:287)
> at org.apache.pig.PigServer.compilePp(PigServer.java:1317)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254)
> at org.apache.pig.PigServer.storeEx(PigServer.java:953)
> ... 14 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2243: Attempt to remove operator LOLimit that is still softly connected in the plan
> at org.apache.pig.newplan.BaseOperatorPlan.remove(BaseOperatorPlan.java:174)
> at org.apache.pig.newplan.BaseOperatorPlan.removeAndReconnect(BaseOperatorPlan.java:449)
> at org.apache.pig.newplan.logical.rules.LimitOptimizer$OptimizeLimitTransformer.transform(LimitOptimizer.java:124)
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2570) LimitOptimizer fails with dynamic
LIMIT argument
Posted by "Daniel Dai (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai resolved PIG-2570.
-----------------------------
Resolution: Fixed
Assignee: Daniel Dai (was: Bill Graham)
Hadoop Flags: Reviewed
test-patch:
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] -1 release audit. The applied patch generated 535 release audit warnings (more than the trunk's current 532 warnings).
javadoc and release audit warning is unrelated.
Patch committed to 0.10/trunk.
> LimitOptimizer fails with dynamic LIMIT argument
> ------------------------------------------------
>
> Key: PIG-2570
> URL: https://issues.apache.org/jira/browse/PIG-2570
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2570-1.patch, PIG-2570-2.patch, data1.txt, limit-fails.pig
>
>
> The following script fails with the exception shown below. Passing {{-t LimitOptimizer}} makes it work, as does adding an {{ORDER}} clause before the limit.
> {noformat}
> A = LOAD 'data1.txt' AS (owner:chararray,pet:chararray,age:int,phone:chararray);
> B = group A all;
> C = foreach B generate SUM(A.age) as total;
> D = foreach A generate owner, age/(double)C.total AS percentAge;
> F = LIMIT D C.total/8;
> DUMP F;
> {noformat}
> {noformat}
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule LimitOptimizer. Try -t LimitOptimizer
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:287)
> at org.apache.pig.PigServer.compilePp(PigServer.java:1317)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254)
> at org.apache.pig.PigServer.storeEx(PigServer.java:953)
> ... 14 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2243: Attempt to remove operator LOLimit that is still softly connected in the plan
> at org.apache.pig.newplan.BaseOperatorPlan.remove(BaseOperatorPlan.java:174)
> at org.apache.pig.newplan.BaseOperatorPlan.removeAndReconnect(BaseOperatorPlan.java:449)
> at org.apache.pig.newplan.logical.rules.LimitOptimizer$OptimizeLimitTransformer.transform(LimitOptimizer.java:124)
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2570) LimitOptimizer fails with dynamic
LIMIT argument
Posted by "Dmitriy V. Ryaboy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228615#comment-13228615 ]
Dmitriy V. Ryaboy commented on PIG-2570:
----------------------------------------
+1, please commit.
> LimitOptimizer fails with dynamic LIMIT argument
> ------------------------------------------------
>
> Key: PIG-2570
> URL: https://issues.apache.org/jira/browse/PIG-2570
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2570-1.patch, PIG-2570-2.patch, data1.txt, limit-fails.pig
>
>
> The following script fails with the exception shown below. Passing {{-t LimitOptimizer}} makes it work, as does adding an {{ORDER}} clause before the limit.
> {noformat}
> A = LOAD 'data1.txt' AS (owner:chararray,pet:chararray,age:int,phone:chararray);
> B = group A all;
> C = foreach B generate SUM(A.age) as total;
> D = foreach A generate owner, age/(double)C.total AS percentAge;
> F = LIMIT D C.total/8;
> DUMP F;
> {noformat}
> {noformat}
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule LimitOptimizer. Try -t LimitOptimizer
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:287)
> at org.apache.pig.PigServer.compilePp(PigServer.java:1317)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254)
> at org.apache.pig.PigServer.storeEx(PigServer.java:953)
> ... 14 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2243: Attempt to remove operator LOLimit that is still softly connected in the plan
> at org.apache.pig.newplan.BaseOperatorPlan.remove(BaseOperatorPlan.java:174)
> at org.apache.pig.newplan.BaseOperatorPlan.removeAndReconnect(BaseOperatorPlan.java:449)
> at org.apache.pig.newplan.logical.rules.LimitOptimizer$OptimizeLimitTransformer.transform(LimitOptimizer.java:124)
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2570) LimitOptimizer fails with dynamic LIMIT
argument
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2570:
----------------------------
Fix Version/s: 0.11
0.10
> LimitOptimizer fails with dynamic LIMIT argument
> ------------------------------------------------
>
> Key: PIG-2570
> URL: https://issues.apache.org/jira/browse/PIG-2570
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2570-1.patch, data1.txt, limit-fails.pig
>
>
> The following script fails with the exception shown below. Passing {{-t LimitOptimizer}} makes it work, as does adding an {{ORDER}} clause before the limit.
> {noformat}
> A = LOAD 'data1.txt' AS (owner:chararray,pet:chararray,age:int,phone:chararray);
> B = group A all;
> C = foreach B generate SUM(A.age) as total;
> D = foreach A generate owner, age/(double)C.total AS percentAge;
> F = LIMIT D C.total/8;
> DUMP F;
> {noformat}
> {noformat}
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule LimitOptimizer. Try -t LimitOptimizer
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:287)
> at org.apache.pig.PigServer.compilePp(PigServer.java:1317)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254)
> at org.apache.pig.PigServer.storeEx(PigServer.java:953)
> ... 14 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2243: Attempt to remove operator LOLimit that is still softly connected in the plan
> at org.apache.pig.newplan.BaseOperatorPlan.remove(BaseOperatorPlan.java:174)
> at org.apache.pig.newplan.BaseOperatorPlan.removeAndReconnect(BaseOperatorPlan.java:449)
> at org.apache.pig.newplan.logical.rules.LimitOptimizer$OptimizeLimitTransformer.transform(LimitOptimizer.java:124)
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2570) LimitOptimizer fails with dynamic LIMIT
argument
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2570:
----------------------------
Attachment: PIG-2570-2.patch
Good catch. Attach PIG-2570-2.patch.
> LimitOptimizer fails with dynamic LIMIT argument
> ------------------------------------------------
>
> Key: PIG-2570
> URL: https://issues.apache.org/jira/browse/PIG-2570
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2570-1.patch, PIG-2570-2.patch, data1.txt, limit-fails.pig
>
>
> The following script fails with the exception shown below. Passing {{-t LimitOptimizer}} makes it work, as does adding an {{ORDER}} clause before the limit.
> {noformat}
> A = LOAD 'data1.txt' AS (owner:chararray,pet:chararray,age:int,phone:chararray);
> B = group A all;
> C = foreach B generate SUM(A.age) as total;
> D = foreach A generate owner, age/(double)C.total AS percentAge;
> F = LIMIT D C.total/8;
> DUMP F;
> {noformat}
> {noformat}
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule LimitOptimizer. Try -t LimitOptimizer
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:287)
> at org.apache.pig.PigServer.compilePp(PigServer.java:1317)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254)
> at org.apache.pig.PigServer.storeEx(PigServer.java:953)
> ... 14 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2243: Attempt to remove operator LOLimit that is still softly connected in the plan
> at org.apache.pig.newplan.BaseOperatorPlan.remove(BaseOperatorPlan.java:174)
> at org.apache.pig.newplan.BaseOperatorPlan.removeAndReconnect(BaseOperatorPlan.java:449)
> at org.apache.pig.newplan.logical.rules.LimitOptimizer$OptimizeLimitTransformer.transform(LimitOptimizer.java:124)
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2570) LimitOptimizer fails with dynamic LIMIT
argument
Posted by "Bill Graham (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated PIG-2570:
-----------------------------
Attachment: limit-fails.pig
data1.txt
Attaching a sample script and datafile to reproduce with.
> LimitOptimizer fails with dynamic LIMIT argument
> ------------------------------------------------
>
> Key: PIG-2570
> URL: https://issues.apache.org/jira/browse/PIG-2570
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: data1.txt, limit-fails.pig
>
>
> The following script fails with the exception shown below. Passing {{-t LimitOptimizer}} makes it work, as does adding an {{ORDER}} clause before the limit.
> {noformat}
> A = LOAD 'data1.txt' AS (owner:chararray,pet:chararray,age:int,phone:chararray);
> B = group A all;
> C = foreach B generate SUM(A.age) as total;
> D = foreach A generate owner, age/(double)C.total AS percentAge;
> F = LIMIT D C.total/8;
> DUMP F;
> {noformat}
> {noformat}
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule LimitOptimizer. Try -t LimitOptimizer
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:287)
> at org.apache.pig.PigServer.compilePp(PigServer.java:1317)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254)
> at org.apache.pig.PigServer.storeEx(PigServer.java:953)
> ... 14 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2243: Attempt to remove operator LOLimit that is still softly connected in the plan
> at org.apache.pig.newplan.BaseOperatorPlan.remove(BaseOperatorPlan.java:174)
> at org.apache.pig.newplan.BaseOperatorPlan.removeAndReconnect(BaseOperatorPlan.java:449)
> at org.apache.pig.newplan.logical.rules.LimitOptimizer$OptimizeLimitTransformer.transform(LimitOptimizer.java:124)
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2570) LimitOptimizer fails with dynamic LIMIT
argument
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2570:
----------------------------
Attachment: PIG-2570-1.patch
> LimitOptimizer fails with dynamic LIMIT argument
> ------------------------------------------------
>
> Key: PIG-2570
> URL: https://issues.apache.org/jira/browse/PIG-2570
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2570-1.patch, data1.txt, limit-fails.pig
>
>
> The following script fails with the exception shown below. Passing {{-t LimitOptimizer}} makes it work, as does adding an {{ORDER}} clause before the limit.
> {noformat}
> A = LOAD 'data1.txt' AS (owner:chararray,pet:chararray,age:int,phone:chararray);
> B = group A all;
> C = foreach B generate SUM(A.age) as total;
> D = foreach A generate owner, age/(double)C.total AS percentAge;
> F = LIMIT D C.total/8;
> DUMP F;
> {noformat}
> {noformat}
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule LimitOptimizer. Try -t LimitOptimizer
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:287)
> at org.apache.pig.PigServer.compilePp(PigServer.java:1317)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254)
> at org.apache.pig.PigServer.storeEx(PigServer.java:953)
> ... 14 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2243: Attempt to remove operator LOLimit that is still softly connected in the plan
> at org.apache.pig.newplan.BaseOperatorPlan.remove(BaseOperatorPlan.java:174)
> at org.apache.pig.newplan.BaseOperatorPlan.removeAndReconnect(BaseOperatorPlan.java:449)
> at org.apache.pig.newplan.logical.rules.LimitOptimizer$OptimizeLimitTransformer.transform(LimitOptimizer.java:124)
> at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira