You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Alex Bain <am...@gmail.com> on 2014/02/10 20:40:45 UTC
Review Request 17914: PIG-3756 Add support for non-Java UDF's in Pig-on-Tez
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17914/
-----------------------------------------------------------
Review request for pig, Cheolsoo Park, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
Bugs: PIG-3756
https://issues.apache.org/jira/browse/PIG-3756
Repository: pig-git
Description
-------
Add support for non-Java UDF's in Pig-on-Tez: Jython, JRuby, JavaScript, Groovy and Streaming UDF's (which is currently cpython). Here is the status of each language with this patch:
Jython -> Works
JRuby -> Works
JavaScript -> Works
Streaming UDF -> Broken due to https://issues.apache.org/jira/browse/PIG-3478. This needs to be fixed before we can make it work in Tez.
Groovy -> Works in Tez when additional changes are made to build.xml and bin/pig (NOT included in my patch, let me know if you want me to add). I think Groovy is broken in MR Pig at the moment and that no one can use it. To make it work in Tez, I have a hack in TezPlanContainer.java that specifically looks for Groovy files. Let me know if you think we should keep this line or perhaps remove it.
Just to note, one way to fix MR Pig would be to add the Groovy files to DistributedCache.
Diffs
-----
src/org/apache/pig/backend/hadoop/executionengine/tez/TezPlanContainer.java 7342dab
src/org/apache/pig/impl/util/JarManager.java 5b36046
src/org/apache/pig/scripting/ScriptEngine.java 182b8f8
Diff: https://reviews.apache.org/r/17914/diff/
Testing
-------
ant test-tez passed
Thanks,
Alex Bain
Re: Review Request 17914: PIG-3756 Add support for non-Java UDF's in
Pig-on-Tez
Posted by Daniel Dai <da...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17914/#review34132
-----------------------------------------------------------
Ship it!
Ship It!
- Daniel Dai
On Feb. 10, 2014, 7:40 p.m., Alex Bain wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17914/
> -----------------------------------------------------------
>
> (Updated Feb. 10, 2014, 7:40 p.m.)
>
>
> Review request for pig, Cheolsoo Park, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3756
> https://issues.apache.org/jira/browse/PIG-3756
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Add support for non-Java UDF's in Pig-on-Tez: Jython, JRuby, JavaScript, Groovy and Streaming UDF's (which is currently cpython). Here is the status of each language with this patch:
>
> Jython -> Works
>
> JRuby -> Works
>
> JavaScript -> Works
>
> Streaming UDF -> Broken due to https://issues.apache.org/jira/browse/PIG-3478. This needs to be fixed before we can make it work in Tez.
>
> Groovy -> Works in Tez when additional changes are made to build.xml and bin/pig (NOT included in my patch, let me know if you want me to add). I think Groovy is broken in MR Pig at the moment and that no one can use it. To make it work in Tez, I have a hack in TezPlanContainer.java that specifically looks for Groovy files. Let me know if you think we should keep this line or perhaps remove it.
>
> Just to note, one way to fix MR Pig would be to add the Groovy files to DistributedCache.
>
>
> Diffs
> -----
>
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezPlanContainer.java 7342dab
> src/org/apache/pig/impl/util/JarManager.java 5b36046
> src/org/apache/pig/scripting/ScriptEngine.java 182b8f8
>
> Diff: https://reviews.apache.org/r/17914/diff/
>
>
> Testing
> -------
>
> ant test-tez passed
>
>
> Thanks,
>
> Alex Bain
>
>