You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Josh Levy (JIRA)" <ji...@apache.org> on 2012/05/16 22:21:09 UTC
[jira] [Created] (PIG-2705) outputSchema modification from
scripting UDFs
Josh Levy created PIG-2705:
------------------------------
Summary: outputSchema modification from scripting UDFs
Key: PIG-2705
URL: https://issues.apache.org/jira/browse/PIG-2705
Project: Pig
Issue Type: Improvement
Affects Versions: 0.10.0, 0.9.2
Reporter: Josh Levy
The examples for @schemaFunction that I've seen on the web (http://pig.apache.org/docs/r0.10.0/udf.html#decorators) tend to be trivial ones that return the same schema passed in. I had a problem where I wrote a python udf that added a new field into a tuple in a bag. I ended up not specifying schema in python and then casting in pig. How does the community feel about making it easier to generate a dynamic schema from a scripting UDF?
This could be done by modifying JythonFunction.outputSchema to handle outputSchemaFunc returning a string like @outputSchema does, or by exposing a public api that makes it easier to manipulate org.apache.pig.impl.logicalLayer.schema.Schema
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2705) outputSchema modification from
scripting UDFs
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Le Dem updated PIG-2705:
-------------------------------
Resolution: Fixed
Fix Version/s: 0.11
Status: Resolved (was: Patch Available)
committed to trunk.
Thanks Josh!
> outputSchema modification from scripting UDFs
> ---------------------------------------------
>
> Key: PIG-2705
> URL: https://issues.apache.org/jira/browse/PIG-2705
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.9.2, 0.10.0
> Reporter: Josh Levy
> Fix For: 0.11
>
> Attachments: PIG-2705-1.patch, PIG-2705.2.patch
>
>
> The examples for @schemaFunction that I've seen on the web (http://pig.apache.org/docs/r0.10.0/udf.html#decorators) tend to be trivial ones that return the same schema passed in. I had a problem where I wrote a python udf that added a new field into a tuple in a bag. I ended up not specifying schema in python and then casting in pig. How does the community feel about making it easier to generate a dynamic schema from a scripting UDF?
> This could be done by modifying JythonFunction.outputSchema to handle outputSchemaFunc returning a string like @outputSchema does, or by exposing a public api that makes it easier to manipulate org.apache.pig.impl.logicalLayer.schema.Schema
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2705) outputSchema modification from
scripting UDFs
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286733#comment-13286733 ]
Julien Le Dem commented on PIG-2705:
------------------------------------
+1 patch looks good to me
> outputSchema modification from scripting UDFs
> ---------------------------------------------
>
> Key: PIG-2705
> URL: https://issues.apache.org/jira/browse/PIG-2705
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.9.2, 0.10.0
> Reporter: Josh Levy
> Attachments: PIG-2705-1.patch, PIG-2705.2.patch
>
>
> The examples for @schemaFunction that I've seen on the web (http://pig.apache.org/docs/r0.10.0/udf.html#decorators) tend to be trivial ones that return the same schema passed in. I had a problem where I wrote a python udf that added a new field into a tuple in a bag. I ended up not specifying schema in python and then casting in pig. How does the community feel about making it easier to generate a dynamic schema from a scripting UDF?
> This could be done by modifying JythonFunction.outputSchema to handle outputSchemaFunc returning a string like @outputSchema does, or by exposing a public api that makes it easier to manipulate org.apache.pig.impl.logicalLayer.schema.Schema
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2705) outputSchema modification from
scripting UDFs
Posted by "Josh Levy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Levy updated PIG-2705:
---------------------------
Attachment: PIG-2705-1.patch
The patch that goes with my comment from 16/May/12 20:26
> outputSchema modification from scripting UDFs
> ---------------------------------------------
>
> Key: PIG-2705
> URL: https://issues.apache.org/jira/browse/PIG-2705
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.9.2, 0.10.0
> Reporter: Josh Levy
> Attachments: PIG-2705-1.patch
>
>
> The examples for @schemaFunction that I've seen on the web (http://pig.apache.org/docs/r0.10.0/udf.html#decorators) tend to be trivial ones that return the same schema passed in. I had a problem where I wrote a python udf that added a new field into a tuple in a bag. I ended up not specifying schema in python and then casting in pig. How does the community feel about making it easier to generate a dynamic schema from a scripting UDF?
> This could be done by modifying JythonFunction.outputSchema to handle outputSchemaFunc returning a string like @outputSchema does, or by exposing a public api that makes it easier to manipulate org.apache.pig.impl.logicalLayer.schema.Schema
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2705) outputSchema modification from
scripting UDFs
Posted by "Josh Levy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Levy updated PIG-2705:
---------------------------
Status: Patch Available (was: Open)
This patch lets the python schemaFunction return a String or an instance of org.apache.pig.ResourceSchema. ResourceSchema feels "more public" than org.apache.pig.impl.logicalLayer.schema.Schema -- maybe it's a better way to programatically manipulate Schema from python
> outputSchema modification from scripting UDFs
> ---------------------------------------------
>
> Key: PIG-2705
> URL: https://issues.apache.org/jira/browse/PIG-2705
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.10.0, 0.9.2
> Reporter: Josh Levy
>
> The examples for @schemaFunction that I've seen on the web (http://pig.apache.org/docs/r0.10.0/udf.html#decorators) tend to be trivial ones that return the same schema passed in. I had a problem where I wrote a python udf that added a new field into a tuple in a bag. I ended up not specifying schema in python and then casting in pig. How does the community feel about making it easier to generate a dynamic schema from a scripting UDF?
> This could be done by modifying JythonFunction.outputSchema to handle outputSchemaFunc returning a string like @outputSchema does, or by exposing a public api that makes it easier to manipulate org.apache.pig.impl.logicalLayer.schema.Schema
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2705) outputSchema modification from
scripting UDFs
Posted by "Josh Levy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Levy updated PIG-2705:
---------------------------
Attachment: PIG-2705.2.patch
Patch reflects Julien Le Dem's comment from 16 May/12
> outputSchema modification from scripting UDFs
> ---------------------------------------------
>
> Key: PIG-2705
> URL: https://issues.apache.org/jira/browse/PIG-2705
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.9.2, 0.10.0
> Reporter: Josh Levy
> Attachments: PIG-2705-1.patch, PIG-2705.2.patch
>
>
> The examples for @schemaFunction that I've seen on the web (http://pig.apache.org/docs/r0.10.0/udf.html#decorators) tend to be trivial ones that return the same schema passed in. I had a problem where I wrote a python udf that added a new field into a tuple in a bag. I ended up not specifying schema in python and then casting in pig. How does the community feel about making it easier to generate a dynamic schema from a scripting UDF?
> This could be done by modifying JythonFunction.outputSchema to handle outputSchemaFunc returning a string like @outputSchema does, or by exposing a public api that makes it easier to manipulate org.apache.pig.impl.logicalLayer.schema.Schema
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2705) outputSchema modification from
scripting UDFs
Posted by "Josh Levy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279176#comment-13279176 ]
Josh Levy commented on PIG-2705:
--------------------------------
Does anyone have feedback about which if any of org.apache.pig.impl.logicalLayer.schema.Schema, org.apache.pig.ResourceSchema, and String manipulation is the "right" way for a UDF author to dynamically create or modify a schema?
> outputSchema modification from scripting UDFs
> ---------------------------------------------
>
> Key: PIG-2705
> URL: https://issues.apache.org/jira/browse/PIG-2705
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.9.2, 0.10.0
> Reporter: Josh Levy
> Attachments: PIG-2705-1.patch, PIG-2705.2.patch
>
>
> The examples for @schemaFunction that I've seen on the web (http://pig.apache.org/docs/r0.10.0/udf.html#decorators) tend to be trivial ones that return the same schema passed in. I had a problem where I wrote a python udf that added a new field into a tuple in a bag. I ended up not specifying schema in python and then casting in pig. How does the community feel about making it easier to generate a dynamic schema from a scripting UDF?
> This could be done by modifying JythonFunction.outputSchema to handle outputSchemaFunc returning a string like @outputSchema does, or by exposing a public api that makes it easier to manipulate org.apache.pig.impl.logicalLayer.schema.Schema
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2705) outputSchema modification from
scripting UDFs
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277277#comment-13277277 ]
Julien Le Dem commented on PIG-2705:
------------------------------------
Hi Josh
Thanks for the contribution.
This looks good to me.
I'd just not reuse the same variable name.
IE: something like that
{code}
Object newSchema = ((pf.__call__(Py.java2py(input))).__tojava__(Object.class));
if (newSchema instanceof ResourceSchema) {
return Schema.getPigSchema((ResourceSchema) newSchema);
} else if (newSchema instanceof Schema) {
return (Schema) newSchema;
} else {
return Utils.getSchemaFromString(newSchema.toString());
}
{code}
> outputSchema modification from scripting UDFs
> ---------------------------------------------
>
> Key: PIG-2705
> URL: https://issues.apache.org/jira/browse/PIG-2705
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.9.2, 0.10.0
> Reporter: Josh Levy
> Attachments: PIG-2705-1.patch
>
>
> The examples for @schemaFunction that I've seen on the web (http://pig.apache.org/docs/r0.10.0/udf.html#decorators) tend to be trivial ones that return the same schema passed in. I had a problem where I wrote a python udf that added a new field into a tuple in a bag. I ended up not specifying schema in python and then casting in pig. How does the community feel about making it easier to generate a dynamic schema from a scripting UDF?
> This could be done by modifying JythonFunction.outputSchema to handle outputSchemaFunc returning a string like @outputSchema does, or by exposing a public api that makes it easier to manipulate org.apache.pig.impl.logicalLayer.schema.Schema
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2705) outputSchema modification from
scripting UDFs
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279991#comment-13279991 ]
Daniel Dai commented on PIG-2705:
---------------------------------
org.apache.pig.impl.logicalLayer.schema.Schema, which we used in EvalFunc. This is the data structure to expose to developer.
> outputSchema modification from scripting UDFs
> ---------------------------------------------
>
> Key: PIG-2705
> URL: https://issues.apache.org/jira/browse/PIG-2705
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.9.2, 0.10.0
> Reporter: Josh Levy
> Attachments: PIG-2705-1.patch, PIG-2705.2.patch
>
>
> The examples for @schemaFunction that I've seen on the web (http://pig.apache.org/docs/r0.10.0/udf.html#decorators) tend to be trivial ones that return the same schema passed in. I had a problem where I wrote a python udf that added a new field into a tuple in a bag. I ended up not specifying schema in python and then casting in pig. How does the community feel about making it easier to generate a dynamic schema from a scripting UDF?
> This could be done by modifying JythonFunction.outputSchema to handle outputSchemaFunc returning a string like @outputSchema does, or by exposing a public api that makes it easier to manipulate org.apache.pig.impl.logicalLayer.schema.Schema
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira