You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Viraj Bhat (JIRA)" <ji...@apache.org> on 2009/04/21 02:17:50 UTC
[jira] Created: (PIG-772) Semantics of Filter statement inside
ForEach should support filtering on aliases used in the Group statement
preeceding it
Semantics of Filter statement inside ForEach should support filtering on aliases used in the Group statement preeceding it
--------------------------------------------------------------------------------------------------------------------------
Key: PIG-772
URL: https://issues.apache.org/jira/browse/PIG-772
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.3.0
Reporter: Viraj Bhat
Priority: Minor
Fix For: 0.3.0
I have a Pig script which tries to display all bags which are greater than the average value in the group.
{code}
A = LOAD 'half.txt' AS (key:CHARARRAY, val:INT);
B = GROUP A BY key;
C = FOREACH B {
N = AVG(A.val);
HALF = FILTER A by val >= N;
GENERATE
FLATTEN(GROUP),
HALF;
};
dump C;
{code}
Presently the semantics of the Filter statement inside the FOREACH does not support these types of operations.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-772) Semantics of Filter statement inside
ForEach should support filtering on aliases used in the Group statement
preceding it
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-772:
---------------------------
Assignee: Alan Gates
Fix Version/s: 0.9.0
> Semantics of Filter statement inside ForEach should support filtering on aliases used in the Group statement preceding it
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-772
> URL: https://issues.apache.org/jira/browse/PIG-772
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.3.0
> Reporter: Viraj Bhat
> Assignee: Alan Gates
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: half.txt
>
>
> I have a Pig script which tries to display all bags which are greater than the average value in the group.
> Input: half.txt
> ===================
> A 1
> A 2
> A 3
> B 1
> B 3
> ====================
> {code}
> A = LOAD 'half.txt' AS (key:CHARARRAY, val:INT);
> B = GROUP A BY key;
> C = FOREACH B {
> N = AVG(A.val);
> HALF = FILTER A by val >= N;
> GENERATE
> FLATTEN(GROUP),
> HALF;
> };
> dump C;
> {code}
> ====================
> Expected Output:
> ====================
> (A,{(A,2),(A,3)})
> (B,{(B,3)})
> ====================
> Presently the semantics of the Filter statement inside the FOREACH does not support these types of operations.
> Error when running the above script.
> =========================================================================================
> ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
> at org.apache.pig.PigServer.parseQuery(PigServer.java:320)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:279)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:529)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:280)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:99)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:88)
> at org.apache.pig.Main.main(Main.java:364)
> =========================================================================================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-772) Semantics of Filter statement inside
ForEach should support filtering on aliases used in the Group statement
preeceding it
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701003#action_12701003 ]
Viraj Bhat commented on PIG-772:
--------------------------------
There seems to be a workaround for the same, but the question is does the below Pig script perform better than the nested Pig script in the original description. In fact there are potentially big performance advantages if the filter statement allowed the semantics in the Original description of this Jira. This will also avoid multiple redundant passes though the data.
{code}
A = LOAD 'half.txt' AS (key:CHARARRAY, val:INT);
B = GROUP A BY key;
C = foreach B { N = AVG(A.val); generate group, flatten(A.val), N as N;};
D = filter C by val >= N;
E = foreach D generate group, val;
F = group E by group;
G = foreach F generate group, E;
dump G
{code}
Input: half.txt
===================
A 1
A 2
A 3
B 1
B 3
====================
Result:
====================
(A,{(A,2),(A,3)})
(B,{(B,3)})
====================
> Semantics of Filter statement inside ForEach should support filtering on aliases used in the Group statement preeceding it
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-772
> URL: https://issues.apache.org/jira/browse/PIG-772
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.3.0
> Reporter: Viraj Bhat
> Priority: Minor
> Fix For: 0.3.0
>
>
> I have a Pig script which tries to display all bags which are greater than the average value in the group.
> {code}
> A = LOAD 'half.txt' AS (key:CHARARRAY, val:INT);
> B = GROUP A BY key;
> C = FOREACH B {
> N = AVG(A.val);
> HALF = FILTER A by val >= N;
> GENERATE
> FLATTEN(GROUP),
> HALF;
> };
> dump C;
> {code}
> Presently the semantics of the Filter statement inside the FOREACH does not support these types of operations.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-772) Semantics of Filter statement inside
ForEach should support filtering on aliases used in the Group statement
preceding it
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Bhat updated PIG-772:
---------------------------
Attachment: half.txt
Input file
> Semantics of Filter statement inside ForEach should support filtering on aliases used in the Group statement preceding it
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-772
> URL: https://issues.apache.org/jira/browse/PIG-772
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.3.0
> Reporter: Viraj Bhat
> Priority: Minor
> Fix For: 0.3.0
>
> Attachments: half.txt
>
>
> I have a Pig script which tries to display all bags which are greater than the average value in the group.
> Input: half.txt
> ===================
> A 1
> A 2
> A 3
> B 1
> B 3
> ====================
> {code}
> A = LOAD 'half.txt' AS (key:CHARARRAY, val:INT);
> B = GROUP A BY key;
> C = FOREACH B {
> N = AVG(A.val);
> HALF = FILTER A by val >= N;
> GENERATE
> FLATTEN(GROUP),
> HALF;
> };
> dump C;
> {code}
> ====================
> Expected Output:
> ====================
> (A,{(A,2),(A,3)})
> (B,{(B,3)})
> ====================
> Presently the semantics of the Filter statement inside the FOREACH does not support these types of operations.
> Error when running the above script.
> =========================================================================================
> ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
> at org.apache.pig.PigServer.parseQuery(PigServer.java:320)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:279)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:529)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:280)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:99)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:88)
> at org.apache.pig.Main.main(Main.java:364)
> =========================================================================================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-772) Semantics of Filter statement inside
ForEach should support filtering on aliases used in the Group statement
preeceding it
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Bhat updated PIG-772:
---------------------------
Issue Type: Improvement (was: Bug)
> Semantics of Filter statement inside ForEach should support filtering on aliases used in the Group statement preeceding it
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-772
> URL: https://issues.apache.org/jira/browse/PIG-772
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.3.0
> Reporter: Viraj Bhat
> Priority: Minor
> Fix For: 0.3.0
>
>
> I have a Pig script which tries to display all bags which are greater than the average value in the group.
> Input: half.txt
> ===================
> A 1
> A 2
> A 3
> B 1
> B 3
> ====================
> {code}
> A = LOAD 'half.txt' AS (key:CHARARRAY, val:INT);
> B = GROUP A BY key;
> C = FOREACH B {
> N = AVG(A.val);
> HALF = FILTER A by val >= N;
> GENERATE
> FLATTEN(GROUP),
> HALF;
> };
> dump C;
> {code}
> ====================
> Expected Output:
> ====================
> (A,{(A,2),(A,3)})
> (B,{(B,3)})
> ====================
> Presently the semantics of the Filter statement inside the FOREACH does not support these types of operations.
> Error when running the above script.
> =========================================================================================
> ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
> at org.apache.pig.PigServer.parseQuery(PigServer.java:320)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:279)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:529)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:280)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:99)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:88)
> at org.apache.pig.Main.main(Main.java:364)
> =========================================================================================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-772) Semantics of Filter statement inside
ForEach should support filtering on aliases used in the Group statement
preceding it
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates resolved PIG-772.
----------------------------
Resolution: Invalid
The error message here is bad, but this is an error. You are trying to secretly do a join in the filter line by referencing two relations (N and A). Pig does not allow a filter operator to have multiple inputs.
> Semantics of Filter statement inside ForEach should support filtering on aliases used in the Group statement preceding it
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-772
> URL: https://issues.apache.org/jira/browse/PIG-772
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.3.0
> Reporter: Viraj Bhat
> Assignee: Alan Gates
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: half.txt
>
>
> I have a Pig script which tries to display all bags which are greater than the average value in the group.
> Input: half.txt
> ===================
> A 1
> A 2
> A 3
> B 1
> B 3
> ====================
> {code}
> A = LOAD 'half.txt' AS (key:CHARARRAY, val:INT);
> B = GROUP A BY key;
> C = FOREACH B {
> N = AVG(A.val);
> HALF = FILTER A by val >= N;
> GENERATE
> FLATTEN(GROUP),
> HALF;
> };
> dump C;
> {code}
> ====================
> Expected Output:
> ====================
> (A,{(A,2),(A,3)})
> (B,{(B,3)})
> ====================
> Presently the semantics of the Filter statement inside the FOREACH does not support these types of operations.
> Error when running the above script.
> =========================================================================================
> ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
> at org.apache.pig.PigServer.parseQuery(PigServer.java:320)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:279)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:529)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:280)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:99)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:88)
> at org.apache.pig.Main.main(Main.java:364)
> =========================================================================================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-772) Semantics of Filter statement inside
ForEach should support filtering on aliases used in the Group statement
preceding it
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Bhat updated PIG-772:
---------------------------
Summary: Semantics of Filter statement inside ForEach should support filtering on aliases used in the Group statement preceding it (was: Semantics of Filter statement inside ForEach should support filtering on aliases used in the Group statement preeceding it)
> Semantics of Filter statement inside ForEach should support filtering on aliases used in the Group statement preceding it
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-772
> URL: https://issues.apache.org/jira/browse/PIG-772
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.3.0
> Reporter: Viraj Bhat
> Priority: Minor
> Fix For: 0.3.0
>
>
> I have a Pig script which tries to display all bags which are greater than the average value in the group.
> Input: half.txt
> ===================
> A 1
> A 2
> A 3
> B 1
> B 3
> ====================
> {code}
> A = LOAD 'half.txt' AS (key:CHARARRAY, val:INT);
> B = GROUP A BY key;
> C = FOREACH B {
> N = AVG(A.val);
> HALF = FILTER A by val >= N;
> GENERATE
> FLATTEN(GROUP),
> HALF;
> };
> dump C;
> {code}
> ====================
> Expected Output:
> ====================
> (A,{(A,2),(A,3)})
> (B,{(B,3)})
> ====================
> Presently the semantics of the Filter statement inside the FOREACH does not support these types of operations.
> Error when running the above script.
> =========================================================================================
> ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
> at org.apache.pig.PigServer.parseQuery(PigServer.java:320)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:279)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:529)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:280)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:99)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:88)
> at org.apache.pig.Main.main(Main.java:364)
> =========================================================================================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-772) Semantics of Filter statement inside
ForEach should support filtering on aliases used in the Group statement
preeceding it
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Bhat updated PIG-772:
---------------------------
Description:
I have a Pig script which tries to display all bags which are greater than the average value in the group.
Input: half.txt
===================
A 1
A 2
A 3
B 1
B 3
====================
{code}
A = LOAD 'half.txt' AS (key:CHARARRAY, val:INT);
B = GROUP A BY key;
C = FOREACH B {
N = AVG(A.val);
HALF = FILTER A by val >= N;
GENERATE
FLATTEN(GROUP),
HALF;
};
dump C;
{code}
====================
Expected Output:
====================
(A,{(A,2),(A,3)})
(B,{(B,3)})
====================
Presently the semantics of the Filter statement inside the FOREACH does not support these types of operations.
Error when running the above script.
=========================================================================================
ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
at org.apache.pig.PigServer.parseQuery(PigServer.java:320)
at org.apache.pig.PigServer.registerQuery(PigServer.java:279)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:529)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:280)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:99)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:88)
at org.apache.pig.Main.main(Main.java:364)
=========================================================================================
was:
I have a Pig script which tries to display all bags which are greater than the average value in the group.
{code}
A = LOAD 'half.txt' AS (key:CHARARRAY, val:INT);
B = GROUP A BY key;
C = FOREACH B {
N = AVG(A.val);
HALF = FILTER A by val >= N;
GENERATE
FLATTEN(GROUP),
HALF;
};
dump C;
{code}
Presently the semantics of the Filter statement inside the FOREACH does not support these types of operations.
> Semantics of Filter statement inside ForEach should support filtering on aliases used in the Group statement preeceding it
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-772
> URL: https://issues.apache.org/jira/browse/PIG-772
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.3.0
> Reporter: Viraj Bhat
> Priority: Minor
> Fix For: 0.3.0
>
>
> I have a Pig script which tries to display all bags which are greater than the average value in the group.
> Input: half.txt
> ===================
> A 1
> A 2
> A 3
> B 1
> B 3
> ====================
> {code}
> A = LOAD 'half.txt' AS (key:CHARARRAY, val:INT);
> B = GROUP A BY key;
> C = FOREACH B {
> N = AVG(A.val);
> HALF = FILTER A by val >= N;
> GENERATE
> FLATTEN(GROUP),
> HALF;
> };
> dump C;
> {code}
> ====================
> Expected Output:
> ====================
> (A,{(A,2),(A,3)})
> (B,{(B,3)})
> ====================
> Presently the semantics of the Filter statement inside the FOREACH does not support these types of operations.
> Error when running the above script.
> =========================================================================================
> ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Invalid alias: A in {key: chararray,val: int}
> at org.apache.pig.PigServer.parseQuery(PigServer.java:320)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:279)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:529)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:280)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:99)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:88)
> at org.apache.pig.Main.main(Main.java:364)
> =========================================================================================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.