You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Jeff Zhang (JIRA)" <ji...@apache.org> on 2009/11/18 04:02:39 UTC
[jira] Created: (PIG-1097) Pig do not support group by boolean type
Pig do not support group by boolean type
----------------------------------------
Key: PIG-1097
URL: https://issues.apache.org/jira/browse/PIG-1097
Project: Pig
Issue Type: Improvement
Components: impl
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Priority: Minor
Fix For: 0.6.0
My Script is as following, the TestUDF return boolean type.
{color:blue}
DEFINE testUDF org.apache.pig.piggybank.util.TestUDF();
raw = LOAD 'data/input';
raw = FOREACH raw GENERATE testUDF();
raw = GROUP raw BY $0;
DUMP raw;
{color}
*The above script will throw exception:*
Exception in thread "main" org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias raw
at org.apache.pig.PigServer.openIterator(PigServer.java:481)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
at org.apache.pig.PigServer.registerScript(PigServer.java:409)
at PigExample.main(PigExample.java:13)
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias raw
at org.apache.pig.PigServer.store(PigServer.java:536)
at org.apache.pig.PigServer.openIterator(PigServer.java:464)
... 5 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:269)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:780)
at org.apache.pig.PigServer.store(PigServer.java:528)
... 6 more
Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2036: Unhandled key type boolean
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.selectComparator(JobControlCompiler.java:856)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:561)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:251)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:128)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:249)
... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1097) Pig do not support group by boolean
type
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828396#action_12828396 ]
Dmitriy V. Ryaboy commented on PIG-1097:
----------------------------------------
Jeff, I see this ticket is assigned to you -- any progress so far? Need a hand?
> Pig do not support group by boolean type
> ----------------------------------------
>
> Key: PIG-1097
> URL: https://issues.apache.org/jira/browse/PIG-1097
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Priority: Minor
>
> My Script is as following, the TestUDF return boolean type.
> {color:blue}
> DEFINE testUDF org.apache.pig.piggybank.util.TestUDF();
> raw = LOAD 'data/input';
> raw = FOREACH raw GENERATE testUDF();
> raw = GROUP raw BY $0;
> DUMP raw;
> {color}
> *The above script will throw exception:*
> Exception in thread "main" org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias raw
> at org.apache.pig.PigServer.openIterator(PigServer.java:481)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
> at org.apache.pig.PigServer.registerScript(PigServer.java:409)
> at PigExample.main(PigExample.java:13)
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias raw
> at org.apache.pig.PigServer.store(PigServer.java:536)
> at org.apache.pig.PigServer.openIterator(PigServer.java:464)
> ... 5 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:269)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:780)
> at org.apache.pig.PigServer.store(PigServer.java:528)
> ... 6 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2036: Unhandled key type boolean
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.selectComparator(JobControlCompiler.java:856)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:561)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:251)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:128)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:249)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1097) Pig do not support group by boolean
type
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779569#action_12779569 ]
Pradeep Kamath commented on PIG-1097:
-------------------------------------
There is a bigger issue here - pig does not support boolean type anywhere except for conditions like in a filter or split. For example, boolean is not allowed in a foreach projection either. We would need to decide if we want to support a boolean type and the implication are more widespread aside from group by.
> Pig do not support group by boolean type
> ----------------------------------------
>
> Key: PIG-1097
> URL: https://issues.apache.org/jira/browse/PIG-1097
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Priority: Minor
> Fix For: 0.6.0
>
>
> My Script is as following, the TestUDF return boolean type.
> {color:blue}
> DEFINE testUDF org.apache.pig.piggybank.util.TestUDF();
> raw = LOAD 'data/input';
> raw = FOREACH raw GENERATE testUDF();
> raw = GROUP raw BY $0;
> DUMP raw;
> {color}
> *The above script will throw exception:*
> Exception in thread "main" org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias raw
> at org.apache.pig.PigServer.openIterator(PigServer.java:481)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
> at org.apache.pig.PigServer.registerScript(PigServer.java:409)
> at PigExample.main(PigExample.java:13)
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias raw
> at org.apache.pig.PigServer.store(PigServer.java:536)
> at org.apache.pig.PigServer.openIterator(PigServer.java:464)
> ... 5 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:269)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:780)
> at org.apache.pig.PigServer.store(PigServer.java:528)
> ... 6 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2036: Unhandled key type boolean
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.selectComparator(JobControlCompiler.java:856)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:561)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:251)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:128)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:249)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1097) Pig do not support group by boolean
type
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780665#action_12780665 ]
Alan Gates commented on PIG-1097:
---------------------------------
FilterFunc did come before types. Before we could deprecate it we need to make Boolean a full fledged type.
I think making Boolean a full type is fine, we just didn't do it when we added types. There's a fair amount of work to do to make it happen. The parser needs to change to support boolean (or bool, whichever we use) and true and false as a keywords. It also needs to change to allow expressions in foreach to be of type boolean. LoadFunc interface needs to change to have a byteToBoolean method. DataReaderWriter can already handle booleans, so no changes there. Physical operators such as =, !=, and is null need to change to support it.
> Pig do not support group by boolean type
> ----------------------------------------
>
> Key: PIG-1097
> URL: https://issues.apache.org/jira/browse/PIG-1097
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Priority: Minor
> Fix For: 0.6.0
>
>
> My Script is as following, the TestUDF return boolean type.
> {color:blue}
> DEFINE testUDF org.apache.pig.piggybank.util.TestUDF();
> raw = LOAD 'data/input';
> raw = FOREACH raw GENERATE testUDF();
> raw = GROUP raw BY $0;
> DUMP raw;
> {color}
> *The above script will throw exception:*
> Exception in thread "main" org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias raw
> at org.apache.pig.PigServer.openIterator(PigServer.java:481)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
> at org.apache.pig.PigServer.registerScript(PigServer.java:409)
> at PigExample.main(PigExample.java:13)
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias raw
> at org.apache.pig.PigServer.store(PigServer.java:536)
> at org.apache.pig.PigServer.openIterator(PigServer.java:464)
> ... 5 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:269)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:780)
> at org.apache.pig.PigServer.store(PigServer.java:528)
> ... 6 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2036: Unhandled key type boolean
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.selectComparator(JobControlCompiler.java:856)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:561)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:251)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:128)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:249)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1097) Pig do not support group by boolean
type
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780380#action_12780380 ]
Jeff Zhang commented on PIG-1097:
---------------------------------
agree, FilterFunc is equivalent to EvalFunc<Boolean> in my opinion. I do not know about the history of FilterFunc, does it come before pig support types? But now I think it should be deprecated.
And why pig do not support boolean type in foreach projection and group by ? any performance consideration ?
> Pig do not support group by boolean type
> ----------------------------------------
>
> Key: PIG-1097
> URL: https://issues.apache.org/jira/browse/PIG-1097
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Priority: Minor
> Fix For: 0.6.0
>
>
> My Script is as following, the TestUDF return boolean type.
> {color:blue}
> DEFINE testUDF org.apache.pig.piggybank.util.TestUDF();
> raw = LOAD 'data/input';
> raw = FOREACH raw GENERATE testUDF();
> raw = GROUP raw BY $0;
> DUMP raw;
> {color}
> *The above script will throw exception:*
> Exception in thread "main" org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias raw
> at org.apache.pig.PigServer.openIterator(PigServer.java:481)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
> at org.apache.pig.PigServer.registerScript(PigServer.java:409)
> at PigExample.main(PigExample.java:13)
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias raw
> at org.apache.pig.PigServer.store(PigServer.java:536)
> at org.apache.pig.PigServer.openIterator(PigServer.java:464)
> ... 5 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:269)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:780)
> at org.apache.pig.PigServer.store(PigServer.java:528)
> ... 6 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2036: Unhandled key type boolean
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.selectComparator(JobControlCompiler.java:856)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:561)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:251)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:128)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:249)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1097) Pig do not support group by boolean type
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1097:
--------------------------------
Fix Version/s: (was: 0.6.0)
> Pig do not support group by boolean type
> ----------------------------------------
>
> Key: PIG-1097
> URL: https://issues.apache.org/jira/browse/PIG-1097
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Priority: Minor
>
> My Script is as following, the TestUDF return boolean type.
> {color:blue}
> DEFINE testUDF org.apache.pig.piggybank.util.TestUDF();
> raw = LOAD 'data/input';
> raw = FOREACH raw GENERATE testUDF();
> raw = GROUP raw BY $0;
> DUMP raw;
> {color}
> *The above script will throw exception:*
> Exception in thread "main" org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias raw
> at org.apache.pig.PigServer.openIterator(PigServer.java:481)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
> at org.apache.pig.PigServer.registerScript(PigServer.java:409)
> at PigExample.main(PigExample.java:13)
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias raw
> at org.apache.pig.PigServer.store(PigServer.java:536)
> at org.apache.pig.PigServer.openIterator(PigServer.java:464)
> ... 5 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:269)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:780)
> at org.apache.pig.PigServer.store(PigServer.java:528)
> ... 6 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2036: Unhandled key type boolean
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.selectComparator(JobControlCompiler.java:856)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:561)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:251)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:128)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:249)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1097) Pig do not support group by boolean
type
Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780223#action_12780223 ]
David Ciemiewicz commented on PIG-1097:
---------------------------------------
I think that one could argue that Filter functions are REALLY just Eval<Boolean> functions in disguise.
That Filter functions were a way of adding return type to Pig for Boolean cases when Pig had no types.
Further, I'd argue, that now that Pig does have data types, that Filter should be deprecated and all Filter functions should now become Eval<Boolean>.
In otherwords, I believe it was an oversight in the types migration to not migrate Filter to Eval<Boolean>
> Pig do not support group by boolean type
> ----------------------------------------
>
> Key: PIG-1097
> URL: https://issues.apache.org/jira/browse/PIG-1097
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Priority: Minor
> Fix For: 0.6.0
>
>
> My Script is as following, the TestUDF return boolean type.
> {color:blue}
> DEFINE testUDF org.apache.pig.piggybank.util.TestUDF();
> raw = LOAD 'data/input';
> raw = FOREACH raw GENERATE testUDF();
> raw = GROUP raw BY $0;
> DUMP raw;
> {color}
> *The above script will throw exception:*
> Exception in thread "main" org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias raw
> at org.apache.pig.PigServer.openIterator(PigServer.java:481)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
> at org.apache.pig.PigServer.registerScript(PigServer.java:409)
> at PigExample.main(PigExample.java:13)
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias raw
> at org.apache.pig.PigServer.store(PigServer.java:536)
> at org.apache.pig.PigServer.openIterator(PigServer.java:464)
> ... 5 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:269)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:780)
> at org.apache.pig.PigServer.store(PigServer.java:528)
> ... 6 more
> Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2036: Unhandled key type boolean
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.selectComparator(JobControlCompiler.java:856)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:561)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:251)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:128)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:249)
> ... 8 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.