You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Santhosh Srinivasan (JIRA)" <ji...@apache.org> on 2008/07/16 23:25:31 UTC
[jira] Created: (PIG-320) The parser/type checker should use the
getSchema method of UDFs to deduce return type/schema
The parser/type checker should use the getSchema method of UDFs to deduce return type/schema
--------------------------------------------------------------------------------------------
Key: PIG-320
URL: https://issues.apache.org/jira/browse/PIG-320
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: types_branch
Reporter: Santhosh Srinivasan
Assignee: Santhosh Srinivasan
Fix For: types_branch
Currently, the parser/type checker uses the getReturnType to deduce the return type of the user defined function (UDF). This mechanism is satisfactory only for basic types (int, long, ...); for composite types (tuple, bag), the schema is also required.The abstract class EvalFunc interface exposes the outputSchema to deduce the return type/schema of the UDF. The parser/type checker should use this method to figure out the return type/schema of the UDF and use it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-320) The parser/type checker should use the
getSchema method of UDFs to deduce return type/schema
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-320:
---------------------------
Status: Patch Available (was: Open)
> The parser/type checker should use the getSchema method of UDFs to deduce return type/schema
> --------------------------------------------------------------------------------------------
>
> Key: PIG-320
> URL: https://issues.apache.org/jira/browse/PIG-320
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Assignee: Santhosh Srinivasan
> Fix For: types_branch
>
> Attachments: udf_outputSchema.patch
>
>
> Currently, the parser/type checker uses the getReturnType to deduce the return type of the user defined function (UDF). This mechanism is satisfactory only for basic types (int, long, ...); for composite types (tuple, bag), the schema is also required.The abstract class EvalFunc interface exposes the outputSchema to deduce the return type/schema of the UDF. The parser/type checker should use this method to figure out the return type/schema of the UDF and use it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-320) The parser/type checker should use the
getSchema method of UDFs to deduce return type/schema
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12616229#action_12616229 ]
Santhosh Srinivasan commented on PIG-320:
-----------------------------------------
The interface outputSchema in EvalFunc has remained unchanged from trunk. Prior to submitting the patch, I tried to change the interface and found that it broke a whole bunch of existing UDFs. There are two things that require a change:
1. The outputSchema method should not take any inputs. It should be outputSchema()
2. The return type of the method should be a FieldSchema and not a Schema
If we decide that these changes are required, I will go ahead and make these changes.
> The parser/type checker should use the getSchema method of UDFs to deduce return type/schema
> --------------------------------------------------------------------------------------------
>
> Key: PIG-320
> URL: https://issues.apache.org/jira/browse/PIG-320
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Assignee: Santhosh Srinivasan
> Fix For: types_branch
>
> Attachments: udf_outputSchema.patch
>
>
> Currently, the parser/type checker uses the getReturnType to deduce the return type of the user defined function (UDF). This mechanism is satisfactory only for basic types (int, long, ...); for composite types (tuple, bag), the schema is also required.The abstract class EvalFunc interface exposes the outputSchema to deduce the return type/schema of the UDF. The parser/type checker should use this method to figure out the return type/schema of the UDF and use it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-320) The parser/type checker should use the
getSchema method of UDFs to deduce return type/schema
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617402#action_12617402 ]
Pi Song commented on PIG-320:
-----------------------------
Since we don't know how EvalFunc is expected to be used. You choice of doing it at outputSchema() is one way to do and doesn't bad.
I agree with the patch.
Please note that EvalFunc should be consolidated at some stage.
> The parser/type checker should use the getSchema method of UDFs to deduce return type/schema
> --------------------------------------------------------------------------------------------
>
> Key: PIG-320
> URL: https://issues.apache.org/jira/browse/PIG-320
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Assignee: Santhosh Srinivasan
> Fix For: types_branch
>
> Attachments: udf_outputSchema.patch
>
>
> Currently, the parser/type checker uses the getReturnType to deduce the return type of the user defined function (UDF). This mechanism is satisfactory only for basic types (int, long, ...); for composite types (tuple, bag), the schema is also required.The abstract class EvalFunc interface exposes the outputSchema to deduce the return type/schema of the UDF. The parser/type checker should use this method to figure out the return type/schema of the UDF and use it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-320) The parser/type checker should use the
getSchema method of UDFs to deduce return type/schema
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614769#action_12614769 ]
Santhosh Srinivasan commented on PIG-320:
-----------------------------------------
EvalFunc implements the outputSchema method and returns null. UDF writers need not implement the outputSchema method. In th typechecker code, we verify the return value of outputSchema method and if it is NULL, then the return type is set to the type returned by getReturnType().
{code}
//In private void setUdfSchema
if (null != udfSchema) {
...
} else {
func.setType(DataType.findType(ef.getReturnType()));
}
{code}
> The parser/type checker should use the getSchema method of UDFs to deduce return type/schema
> --------------------------------------------------------------------------------------------
>
> Key: PIG-320
> URL: https://issues.apache.org/jira/browse/PIG-320
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Assignee: Santhosh Srinivasan
> Fix For: types_branch
>
> Attachments: udf_outputSchema.patch
>
>
> Currently, the parser/type checker uses the getReturnType to deduce the return type of the user defined function (UDF). This mechanism is satisfactory only for basic types (int, long, ...); for composite types (tuple, bag), the schema is also required.The abstract class EvalFunc interface exposes the outputSchema to deduce the return type/schema of the UDF. The parser/type checker should use this method to figure out the return type/schema of the UDF and use it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-320) The parser/type checker should use the
getSchema method of UDFs to deduce return type/schema
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615247#action_12615247 ]
Pi Song commented on PIG-320:
-----------------------------
I'm curios about this.
Then if the outputSchema implementation is provided, why do extract only the first field? Isn't it more intuitive to use the whole schema directly? My suggestion might break some stuffs but it seems a bit better for users.
{noformat}
Schema udfSchema = ef.outputSchema(inputSchema);
if (null != udfSchema) {
Schema.FieldSchema fs;
try {
fs = udfSchema.getField(0);
} catch (ParseException pe) {
throw new VisitorException(pe.getMessage());
}
func.setType(fs.type);
try {
func.setFieldSchema(fs);
} catch (FrontendException fe) {
throw new VisitorException(fe.getMessage());
}
} else {
func.setType(DataType.findType(ef.getReturnType()));
}
{noformat}
> The parser/type checker should use the getSchema method of UDFs to deduce return type/schema
> --------------------------------------------------------------------------------------------
>
> Key: PIG-320
> URL: https://issues.apache.org/jira/browse/PIG-320
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Assignee: Santhosh Srinivasan
> Fix For: types_branch
>
> Attachments: udf_outputSchema.patch
>
>
> Currently, the parser/type checker uses the getReturnType to deduce the return type of the user defined function (UDF). This mechanism is satisfactory only for basic types (int, long, ...); for composite types (tuple, bag), the schema is also required.The abstract class EvalFunc interface exposes the outputSchema to deduce the return type/schema of the UDF. The parser/type checker should use this method to figure out the return type/schema of the UDF and use it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-320) The parser/type checker should use the
getSchema method of UDFs to deduce return type/schema
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Santhosh Srinivasan updated PIG-320:
------------------------------------
Attachment: udf_outputSchema.patch
The patch udf_outputSchema.patch adds the following:
1. The type checker uses the outputSchema method in EvalFunc to deduce the output schema of the UDF
2. The TestEvalPipeline.java:testBagFunctionWithFlattening passes the unit test cases with the addition of outputSchema method to the UDFs used inside the test case
3. Adds resetSchema method to relational operators that rely on nested plans to compute schemas.
4. Resets schema of all the relational operators in the type checker to ensure correctness of the computed schema due to changes in the inputs of the operators.
Unit test cases that still fail are:
[junit] Running org.apache.pig.test.TestFilterOpNumeric
[junit] Tests run: 8, Failures: 0, Errors: 1, Time elapsed: 56.735 sec
[junit] Test org.apache.pig.test.TestFilterOpNumeric FAILED
[junit] Running org.apache.pig.test.TestStoreOld
[junit] Tests run: 3, Failures: 0, Errors: 2, Time elapsed: 21.46 sec
[junit] Test org.apache.pig.test.TestStoreOld FAILED
> The parser/type checker should use the getSchema method of UDFs to deduce return type/schema
> --------------------------------------------------------------------------------------------
>
> Key: PIG-320
> URL: https://issues.apache.org/jira/browse/PIG-320
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Assignee: Santhosh Srinivasan
> Fix For: types_branch
>
> Attachments: udf_outputSchema.patch
>
>
> Currently, the parser/type checker uses the getReturnType to deduce the return type of the user defined function (UDF). This mechanism is satisfactory only for basic types (int, long, ...); for composite types (tuple, bag), the schema is also required.The abstract class EvalFunc interface exposes the outputSchema to deduce the return type/schema of the UDF. The parser/type checker should use this method to figure out the return type/schema of the UDF and use it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-320) The parser/type checker should use the
getSchema method of UDFs to deduce return type/schema
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12616461#action_12616461 ]
Pi Song commented on PIG-320:
-----------------------------
EvalFunc interface is a bit confusing.
Let's wait for clarification from the mailing-list first.
> The parser/type checker should use the getSchema method of UDFs to deduce return type/schema
> --------------------------------------------------------------------------------------------
>
> Key: PIG-320
> URL: https://issues.apache.org/jira/browse/PIG-320
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Assignee: Santhosh Srinivasan
> Fix For: types_branch
>
> Attachments: udf_outputSchema.patch
>
>
> Currently, the parser/type checker uses the getReturnType to deduce the return type of the user defined function (UDF). This mechanism is satisfactory only for basic types (int, long, ...); for composite types (tuple, bag), the schema is also required.The abstract class EvalFunc interface exposes the outputSchema to deduce the return type/schema of the UDF. The parser/type checker should use this method to figure out the return type/schema of the UDF and use it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-320) The parser/type checker should use the
getSchema method of UDFs to deduce return type/schema
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615445#action_12615445 ]
Santhosh Srinivasan commented on PIG-320:
-----------------------------------------
UDFs are expression operators that are expected to return a FieldSchema. The schema returned by outputSchema should always have a single FieldSchema inside. Hence, only the first field is extracted.
> The parser/type checker should use the getSchema method of UDFs to deduce return type/schema
> --------------------------------------------------------------------------------------------
>
> Key: PIG-320
> URL: https://issues.apache.org/jira/browse/PIG-320
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Assignee: Santhosh Srinivasan
> Fix For: types_branch
>
> Attachments: udf_outputSchema.patch
>
>
> Currently, the parser/type checker uses the getReturnType to deduce the return type of the user defined function (UDF). This mechanism is satisfactory only for basic types (int, long, ...); for composite types (tuple, bag), the schema is also required.The abstract class EvalFunc interface exposes the outputSchema to deduce the return type/schema of the UDF. The parser/type checker should use this method to figure out the return type/schema of the UDF and use it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-320) The parser/type checker should use the
getSchema method of UDFs to deduce return type/schema
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614751#action_12614751 ]
Pi Song commented on PIG-320:
-----------------------------
First, let's define a concrete EvalFunc semantic:-
{noformat}
EvalFunc: U -> T
{noformat}
T is hardcoded in UDF implementation but in case that T is a complex type then its schema can be determined by U
getReturnType() should return the hardcoded type (e.g. Int, Double, Bag). outputSchema() should return schemas of complex types (always returning null for simple types).
I think this should be simple enough for users to understand. The most common case would be users creating UDF: simpleType -> simpleType , they don't have to override outputSchema().
Some of the current built-in functions will be broken following this semantic but they can be easily fixed.
Agree?
> The parser/type checker should use the getSchema method of UDFs to deduce return type/schema
> --------------------------------------------------------------------------------------------
>
> Key: PIG-320
> URL: https://issues.apache.org/jira/browse/PIG-320
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Assignee: Santhosh Srinivasan
> Fix For: types_branch
>
> Attachments: udf_outputSchema.patch
>
>
> Currently, the parser/type checker uses the getReturnType to deduce the return type of the user defined function (UDF). This mechanism is satisfactory only for basic types (int, long, ...); for composite types (tuple, bag), the schema is also required.The abstract class EvalFunc interface exposes the outputSchema to deduce the return type/schema of the UDF. The parser/type checker should use this method to figure out the return type/schema of the UDF and use it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-320) The parser/type checker should use the
getSchema method of UDFs to deduce return type/schema
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614550#action_12614550 ]
Alan Gates commented on PIG-320:
--------------------------------
Pi, since much of these changes are in the type checker, I'd like to get your feedback on them.
One comment from me. Why do POForeach and POCogroup have resetSchema() in addition to unsetSchema()? Shouldn't their unsetSchema methods do the work that's in resetSchema? Is there ever a case you'll want to unset the foreach's schema with unsetting it's plan's schemas?
> The parser/type checker should use the getSchema method of UDFs to deduce return type/schema
> --------------------------------------------------------------------------------------------
>
> Key: PIG-320
> URL: https://issues.apache.org/jira/browse/PIG-320
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Assignee: Santhosh Srinivasan
> Fix For: types_branch
>
> Attachments: udf_outputSchema.patch
>
>
> Currently, the parser/type checker uses the getReturnType to deduce the return type of the user defined function (UDF). This mechanism is satisfactory only for basic types (int, long, ...); for composite types (tuple, bag), the schema is also required.The abstract class EvalFunc interface exposes the outputSchema to deduce the return type/schema of the UDF. The parser/type checker should use this method to figure out the return type/schema of the UDF and use it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-320) The parser/type checker should use the
getSchema method of UDFs to deduce return type/schema
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-320:
---------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
udf_outputSchema.patch checked in.
> The parser/type checker should use the getSchema method of UDFs to deduce return type/schema
> --------------------------------------------------------------------------------------------
>
> Key: PIG-320
> URL: https://issues.apache.org/jira/browse/PIG-320
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Assignee: Santhosh Srinivasan
> Fix For: types_branch
>
> Attachments: udf_outputSchema.patch
>
>
> Currently, the parser/type checker uses the getReturnType to deduce the return type of the user defined function (UDF). This mechanism is satisfactory only for basic types (int, long, ...); for composite types (tuple, bag), the schema is also required.The abstract class EvalFunc interface exposes the outputSchema to deduce the return type/schema of the UDF. The parser/type checker should use this method to figure out the return type/schema of the UDF and use it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-320) The parser/type checker should use the
getSchema method of UDFs to deduce return type/schema
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615593#action_12615593 ]
Pi Song commented on PIG-320:
-----------------------------
I think it's fine in terms of internal implementation but can't we make it simpler for users by exposing simpler interface?
> The parser/type checker should use the getSchema method of UDFs to deduce return type/schema
> --------------------------------------------------------------------------------------------
>
> Key: PIG-320
> URL: https://issues.apache.org/jira/browse/PIG-320
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Assignee: Santhosh Srinivasan
> Fix For: types_branch
>
> Attachments: udf_outputSchema.patch
>
>
> Currently, the parser/type checker uses the getReturnType to deduce the return type of the user defined function (UDF). This mechanism is satisfactory only for basic types (int, long, ...); for composite types (tuple, bag), the schema is also required.The abstract class EvalFunc interface exposes the outputSchema to deduce the return type/schema of the UDF. The parser/type checker should use this method to figure out the return type/schema of the UDF and use it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.