You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2008/07/25 09:55:31 UTC
[jira] Created: (PIG-339) Limit follow cross/union return wrong
number of records
Limit follow cross/union return wrong number of records
-------------------------------------------------------
Key: PIG-339
URL: https://issues.apache.org/jira/browse/PIG-339
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: types_branch
Reporter: Daniel Dai
Fix For: types_branch
The following script returns double records as expected:
a = load 'a';
b = load 'b';
c = union a, b;
d = cross a, b;
e = limit c 100;
f = limit d 100;
dump e; // return double number of records
dump f; // return double number of records
Seems to be the limit operator in reduce plan is not effective.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-339) Limit follow cross/union return wrong
number of records
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622738#action_12622738 ]
Daniel Dai commented on PIG-339:
--------------------------------
This error is gone after applying [PIG-378|https://issues.apache.org/jira/browse/PIG-378].
> Limit follow cross/union return wrong number of records
> -------------------------------------------------------
>
> Key: PIG-339
> URL: https://issues.apache.org/jira/browse/PIG-339
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Fix For: types_branch
>
> Attachments: PIG-339.patch
>
>
> The following script returns double records as expected:
> a = load 'a';
> b = load 'b';
> c = union a, b;
> d = cross a, b;
> e = limit c 100;
> f = limit d 100;
> dump e; // return double number of records
> dump f; // return double number of records
> Seems to be the limit operator in reduce plan is not effective.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-339) Limit follow cross/union return wrong
number of records
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich resolved PIG-339.
--------------------------------
Resolution: Fixed
Verified that the issue is gone. Thanks Daniel
> Limit follow cross/union return wrong number of records
> -------------------------------------------------------
>
> Key: PIG-339
> URL: https://issues.apache.org/jira/browse/PIG-339
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Fix For: types_branch
>
> Attachments: PIG-339.patch
>
>
> The following script returns double records as expected:
> a = load 'a';
> b = load 'b';
> c = union a, b;
> d = cross a, b;
> e = limit c 100;
> f = limit d 100;
> dump e; // return double number of records
> dump f; // return double number of records
> Seems to be the limit operator in reduce plan is not effective.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-339) Limit follow cross/union return wrong
number of records
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-339:
---------------------------
Status: Patch Available (was: Open)
> Limit follow cross/union return wrong number of records
> -------------------------------------------------------
>
> Key: PIG-339
> URL: https://issues.apache.org/jira/browse/PIG-339
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Fix For: types_branch
>
> Attachments: PIG-339.patch
>
>
> The following script returns double records as expected:
> a = load 'a';
> b = load 'b';
> c = union a, b;
> d = cross a, b;
> e = limit c 100;
> f = limit d 100;
> dump e; // return double number of records
> dump f; // return double number of records
> Seems to be the limit operator in reduce plan is not effective.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-339) Limit follow cross/union return wrong
number of records
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-339:
---------------------------
Attachment: PIG-339.patch
> Limit follow cross/union return wrong number of records
> -------------------------------------------------------
>
> Key: PIG-339
> URL: https://issues.apache.org/jira/browse/PIG-339
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Fix For: types_branch
>
> Attachments: PIG-339.patch
>
>
> The following script returns double records as expected:
> a = load 'a';
> b = load 'b';
> c = union a, b;
> d = cross a, b;
> e = limit c 100;
> f = limit d 100;
> dump e; // return double number of records
> dump f; // return double number of records
> Seems to be the limit operator in reduce plan is not effective.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-339) Limit follow cross/union return wrong
number of records
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-339:
---------------------------
Status: Open (was: Patch Available)
When I run a script like the following:
{code}
a = load 'studenttab10k';
b = load 'votertab10k';
a1 = foreach a generate $0, $1;
b1 = foreach b generate $0, $1;
c = union a1, b1;
d = limit c 100;
store d into 'result';
{code}
I get:
java.io.IOException: Unable to store for alias: 12 [null]
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.LogToPhyTranslationVisitor.visit(LogToPhyTranslationVisitor.java:1010)
at org.apache.pig.impl.logicalLayer.LOLimit.visit(LOLimit.java:76)
at org.apache.pig.impl.logicalLayer.LOLimit.visit(LOLimit.java:10)
at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:245)
at org.apache.pig.PigServer.compilePp(PigServer.java:590)
at org.apache.pig.PigServer.execute(PigServer.java:516)
at org.apache.pig.PigServer.registerQuery(PigServer.java:265)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:425)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:82)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
at org.apache.pig.Main.main(Main.java:302)
Caused by: java.lang.NullPointerException
> Limit follow cross/union return wrong number of records
> -------------------------------------------------------
>
> Key: PIG-339
> URL: https://issues.apache.org/jira/browse/PIG-339
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Fix For: types_branch
>
> Attachments: PIG-339.patch
>
>
> The following script returns double records as expected:
> a = load 'a';
> b = load 'b';
> c = union a, b;
> d = cross a, b;
> e = limit c 100;
> f = limit d 100;
> dump e; // return double number of records
> dump f; // return double number of records
> Seems to be the limit operator in reduce plan is not effective.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.