You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Christopher Olston (JIRA)" <ji...@apache.org> on 2008/12/01 21:25:44 UTC
[jira] Created: (PIG-549) type checking with order-by following
user-defined function
type checking with order-by following user-defined function
-----------------------------------------------------------
Key: PIG-549
URL: https://issues.apache.org/jira/browse/PIG-549
Project: Pig
Issue Type: Bug
Affects Versions: types_branch
Environment: type checker fails here:
A = load ...;
B = foreach A generate UDF1(*), UDF2();
C = order B by $1;
where UDF2() is of type EvalFunc<Integer>.
I tried all sorts of things, including overriding outputSchema() of the UDF to specify Integer, and also adding "as x : int" to the foreach command -- in all cases I get the same error.
Reporter: Christopher Olston
Fix For: types_branch
Exception in thread "main" java.lang.AssertionError: Unsupported root type in LOForEach:LOUserFunc
at org.apache.pig.impl.logicalLayer.validators.TypeCheckingVisitor.visit(TypeCheckingVisitor.java:2267)
at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:121)
at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:40)
at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
at org.apache.pig.impl.plan.PlanValidator.validateSkipCollectException(PlanValidator.java:101)
at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:40)
at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:30)
at org.apache.pig.impl.logicalLayer.validators.LogicalPlanValidationExecutor.validate(LogicalPlanValidationExecutor.java:79)
at org.apache.pig.PigServer.compileLp(PigServer.java:684)
at org.apache.pig.PigServer.compileLp(PigServer.java:655)
at org.apache.pig.PigServer.store(PigServer.java:433)
at org.apache.pig.PigServer.store(PigServer.java:421)
at org.apache.pig.PigServer.openIterator(PigServer.java:384)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-549) type checking with order-by following
user-defined function
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653028#action_12653028 ]
Santhosh Srinivasan commented on PIG-549:
-----------------------------------------
Sure, we should allow that. I will mark this an enhancement request.
> type checking with order-by following user-defined function
> -----------------------------------------------------------
>
> Key: PIG-549
> URL: https://issues.apache.org/jira/browse/PIG-549
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Environment: type checker fails here:
> A = load ...;
> B = foreach A generate UDF1(*), UDF2();
> C = order B by $1;
> where UDF2() is of type EvalFunc<Integer>.
> I tried all sorts of things, including overriding outputSchema() of the UDF to specify Integer, and also adding "as x : int" to the foreach command -- in all cases I get the same error.
> Reporter: Christopher Olston
> Fix For: types_branch
>
>
> Exception in thread "main" java.lang.AssertionError: Unsupported root type in LOForEach:LOUserFunc
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingVisitor.visit(TypeCheckingVisitor.java:2267)
> at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:121)
> at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:40)
> at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.plan.PlanValidator.validateSkipCollectException(PlanValidator.java:101)
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:40)
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:30)
> at org.apache.pig.impl.logicalLayer.validators.LogicalPlanValidationExecutor.validate(LogicalPlanValidationExecutor.java:79)
> at org.apache.pig.PigServer.compileLp(PigServer.java:684)
> at org.apache.pig.PigServer.compileLp(PigServer.java:655)
> at org.apache.pig.PigServer.store(PigServer.java:433)
> at org.apache.pig.PigServer.store(PigServer.java:421)
> at org.apache.pig.PigServer.openIterator(PigServer.java:384)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-549) type checking with order-by following
user-defined function
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Santhosh Srinivasan updated PIG-549:
------------------------------------
Description:
Exception in thread "main" java.lang.AssertionError: Unsupported root type in LOForEach:LOUserFunc
at org.apache.pig.impl.logicalLayer.validators.TypeCheckingVisitor.visit(TypeCheckingVisitor.java:2267)
at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:121)
at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:40)
at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
at org.apache.pig.impl.plan.PlanValidator.validateSkipCollectException(PlanValidator.java:101)
at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:40)
at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:30)
at org.apache.pig.impl.logicalLayer.validators.LogicalPlanValidationExecutor.validate(LogicalPlanValidationExecutor.java:79)
at org.apache.pig.PigServer.compileLp(PigServer.java:684)
at org.apache.pig.PigServer.compileLp(PigServer.java:655)
at org.apache.pig.PigServer.store(PigServer.java:433)
at org.apache.pig.PigServer.store(PigServer.java:421)
at org.apache.pig.PigServer.openIterator(PigServer.java:384)
was:
Exception in thread "main" java.lang.AssertionError: Unsupported root type in LOForEach:LOUserFunc
at org.apache.pig.impl.logicalLayer.validators.TypeCheckingVisitor.visit(TypeCheckingVisitor.java:2267)
at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:121)
at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:40)
at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
at org.apache.pig.impl.plan.PlanValidator.validateSkipCollectException(PlanValidator.java:101)
at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:40)
at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:30)
at org.apache.pig.impl.logicalLayer.validators.LogicalPlanValidationExecutor.validate(LogicalPlanValidationExecutor.java:79)
at org.apache.pig.PigServer.compileLp(PigServer.java:684)
at org.apache.pig.PigServer.compileLp(PigServer.java:655)
at org.apache.pig.PigServer.store(PigServer.java:433)
at org.apache.pig.PigServer.store(PigServer.java:421)
at org.apache.pig.PigServer.openIterator(PigServer.java:384)
Issue Type: Improvement (was: Bug)
> type checking with order-by following user-defined function
> -----------------------------------------------------------
>
> Key: PIG-549
> URL: https://issues.apache.org/jira/browse/PIG-549
> Project: Pig
> Issue Type: Improvement
> Affects Versions: types_branch
> Environment: type checker fails here:
> A = load ...;
> B = foreach A generate UDF1(*), UDF2();
> C = order B by $1;
> where UDF2() is of type EvalFunc<Integer>.
> I tried all sorts of things, including overriding outputSchema() of the UDF to specify Integer, and also adding "as x : int" to the foreach command -- in all cases I get the same error.
> Reporter: Christopher Olston
> Fix For: types_branch
>
>
> Exception in thread "main" java.lang.AssertionError: Unsupported root type in LOForEach:LOUserFunc
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingVisitor.visit(TypeCheckingVisitor.java:2267)
> at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:121)
> at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:40)
> at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.plan.PlanValidator.validateSkipCollectException(PlanValidator.java:101)
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:40)
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:30)
> at org.apache.pig.impl.logicalLayer.validators.LogicalPlanValidationExecutor.validate(LogicalPlanValidationExecutor.java:79)
> at org.apache.pig.PigServer.compileLp(PigServer.java:684)
> at org.apache.pig.PigServer.compileLp(PigServer.java:655)
> at org.apache.pig.PigServer.store(PigServer.java:433)
> at org.apache.pig.PigServer.store(PigServer.java:421)
> at org.apache.pig.PigServer.openIterator(PigServer.java:384)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-549) type checking with order-by following
user-defined function
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652153#action_12652153 ]
Santhosh Srinivasan commented on PIG-549:
-----------------------------------------
AFAIK, Pig does not support zero argument UDFs. In your script, UDF2() is the reason for the type checking error.
> type checking with order-by following user-defined function
> -----------------------------------------------------------
>
> Key: PIG-549
> URL: https://issues.apache.org/jira/browse/PIG-549
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Environment: type checker fails here:
> A = load ...;
> B = foreach A generate UDF1(*), UDF2();
> C = order B by $1;
> where UDF2() is of type EvalFunc<Integer>.
> I tried all sorts of things, including overriding outputSchema() of the UDF to specify Integer, and also adding "as x : int" to the foreach command -- in all cases I get the same error.
> Reporter: Christopher Olston
> Fix For: types_branch
>
>
> Exception in thread "main" java.lang.AssertionError: Unsupported root type in LOForEach:LOUserFunc
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingVisitor.visit(TypeCheckingVisitor.java:2267)
> at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:121)
> at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:40)
> at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.plan.PlanValidator.validateSkipCollectException(PlanValidator.java:101)
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:40)
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:30)
> at org.apache.pig.impl.logicalLayer.validators.LogicalPlanValidationExecutor.validate(LogicalPlanValidationExecutor.java:79)
> at org.apache.pig.PigServer.compileLp(PigServer.java:684)
> at org.apache.pig.PigServer.compileLp(PigServer.java:655)
> at org.apache.pig.PigServer.store(PigServer.java:433)
> at org.apache.pig.PigServer.store(PigServer.java:421)
> at org.apache.pig.PigServer.openIterator(PigServer.java:384)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-549) type checking with order-by following
user-defined function
Posted by "Christopher Olston (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652972#action_12652972 ]
Christopher Olston commented on PIG-549:
----------------------------------------
Why not simply allow zero-argument UDFs everywhere? I don't get why it's even a special case.
For example, I want to add a random field: B = foreach A generate *, Rand();
> type checking with order-by following user-defined function
> -----------------------------------------------------------
>
> Key: PIG-549
> URL: https://issues.apache.org/jira/browse/PIG-549
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Environment: type checker fails here:
> A = load ...;
> B = foreach A generate UDF1(*), UDF2();
> C = order B by $1;
> where UDF2() is of type EvalFunc<Integer>.
> I tried all sorts of things, including overriding outputSchema() of the UDF to specify Integer, and also adding "as x : int" to the foreach command -- in all cases I get the same error.
> Reporter: Christopher Olston
> Fix For: types_branch
>
>
> Exception in thread "main" java.lang.AssertionError: Unsupported root type in LOForEach:LOUserFunc
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingVisitor.visit(TypeCheckingVisitor.java:2267)
> at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:121)
> at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:40)
> at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.plan.PlanValidator.validateSkipCollectException(PlanValidator.java:101)
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:40)
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:30)
> at org.apache.pig.impl.logicalLayer.validators.LogicalPlanValidationExecutor.validate(LogicalPlanValidationExecutor.java:79)
> at org.apache.pig.PigServer.compileLp(PigServer.java:684)
> at org.apache.pig.PigServer.compileLp(PigServer.java:655)
> at org.apache.pig.PigServer.store(PigServer.java:433)
> at org.apache.pig.PigServer.store(PigServer.java:421)
> at org.apache.pig.PigServer.openIterator(PigServer.java:384)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-549) type checking with order-by following
user-defined function
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652916#action_12652916 ]
Santhosh Srinivasan commented on PIG-549:
-----------------------------------------
Wrt my previous comment, Pig does support zero argument UDFs in foreach but they are allowed in other places like Filter, Order by, etc.
> type checking with order-by following user-defined function
> -----------------------------------------------------------
>
> Key: PIG-549
> URL: https://issues.apache.org/jira/browse/PIG-549
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Environment: type checker fails here:
> A = load ...;
> B = foreach A generate UDF1(*), UDF2();
> C = order B by $1;
> where UDF2() is of type EvalFunc<Integer>.
> I tried all sorts of things, including overriding outputSchema() of the UDF to specify Integer, and also adding "as x : int" to the foreach command -- in all cases I get the same error.
> Reporter: Christopher Olston
> Fix For: types_branch
>
>
> Exception in thread "main" java.lang.AssertionError: Unsupported root type in LOForEach:LOUserFunc
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingVisitor.visit(TypeCheckingVisitor.java:2267)
> at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:121)
> at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:40)
> at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.plan.PlanValidator.validateSkipCollectException(PlanValidator.java:101)
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:40)
> at org.apache.pig.impl.logicalLayer.validators.TypeCheckingValidator.validate(TypeCheckingValidator.java:30)
> at org.apache.pig.impl.logicalLayer.validators.LogicalPlanValidationExecutor.validate(LogicalPlanValidationExecutor.java:79)
> at org.apache.pig.PigServer.compileLp(PigServer.java:684)
> at org.apache.pig.PigServer.compileLp(PigServer.java:655)
> at org.apache.pig.PigServer.store(PigServer.java:433)
> at org.apache.pig.PigServer.store(PigServer.java:421)
> at org.apache.pig.PigServer.openIterator(PigServer.java:384)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.