You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Russell Jurney (JIRA)" <ji...@apache.org> on 2012/09/22 17:01:08 UTC
[jira] [Created] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Russell Jurney created PIG-2927:
-----------------------------------
Summary: SHIP and use JRuby gems in JRuby UDFs
Key: PIG-2927
URL: https://issues.apache.org/jira/browse/PIG-2927
Project: Pig
Issue Type: New Feature
Reporter: Russell Jurney
Assignee: Jonathan Coveney
Priority: Critical
It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479567#comment-13479567 ]
Cheolsoo Park commented on PIG-2927:
------------------------------------
Although I am no Ruby expert, I think that Jonathan's patch works well. Here is my test.
1) installed a non-trivial rubygem library (rubygem-json) on the client only and confirmed that it is not installed on any datanode on the cluster.
{code}
/usr/lib/ruby/gems/1.8/gems/json-1.4.6/
{code}
2) wrote a ruby udf that parses json string:
{code}
require 'rubygems'
require 'pigudf'
require 'json'
class Myudfs < PigUdf
outputSchema "result:chararray"
def parseJson input
result = JSON.parse(input)
end
end
{code}
3) wrote a short pig script that loads a jsonstring and calls my ruby udf:
{code}
register 'test.rb' using jruby as myfuncs;
a = load 'json.txt' using PigStorage() as (i:chararray);
b = foreach a generate myfuncs.parseJson(i);
dump b;
{code}
4) got the expected result as follows:
{code:title=input}
{"id":1,"nested":{"value1":"first1","next":{"complex_record":{"id":2,"nested":{"value1":"second1","next":null,"value2":"second2"}}},"value2":"first2"}}
{code}
{code:title=result}
([id#1,nested#{value1=first1, value2=first2, next={complex_record={id=2, nested={value1=second1, value2=second2, next=null}}}}])
{code}
Without Jonathan's patch, I get the following error in the front-end as expected:
{code}
LoadError: no such file to load -- json
require at org/jruby/RubyKernel.java:1042
require at file:/home/cheolsoo/pig-ruby/build/ivy/lib/Pig/jruby-complete-1.6.7.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36
(root) at test.rb:3
2012-10-18 17:09:24,323 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2999: Unexpected internal error. (LoadError) no such file to load -- json
{code}
I also ran the "Scripting" e2e test cases with the patch on a Hadoop-1.0.x cluster, and they all passed. So it seems good to commit to me.
Btw, I wanted to write an e2e test case using rubygems-json, but I realized that rubygems-json is under GPL and can't include in Pig. We should either find another rubygem library that is under the Apache licence or make the test configurable so that it will run only if rubygem-json is installed.
Thanks!
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Fix For: 0.11
>
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476644#comment-13476644 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
It builds! :) Ok now I test.
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Coveney updated PIG-2927:
----------------------------------
Attachment: PIG-2927-0.patch
I just attached a patch that adds gem support. Any script registered as so:
register script.rb using jruby as myfuncs;
will be instantiated locally, and the set of dependencies across scripts will be shipped.
Note:
- JRUBY_HOME must be set on the client side. This is so we know where to find the gems!
- you need to make sure to have a "require 'pigudf'" since it is instantiated on the client side (I could probably fix this, but I think forcing this is desirable for testing of scripts etc)
Russell,
Any chance you can try this out and see if it works?
I'll add tests sometime...
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Attachments: PIG-2927-0.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Coveney updated PIG-2927:
----------------------------------
Attachment: PIG-2927-4.patch
I radically refactored it because I was getting a lot of annoying errors. It could be refactored further (and made a lot cleaner), but would rather have some eyes on it and testing it. It now passes both the unit tests and the e2e tests (though for the e2e tests to pass, JRUBY_HOME has to be set in your environment...).
Would love thoughts. I focused on trying to get it to work robustly instead of making it beautiful. I can refactor to make it beautiful later :)
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Fix For: 0.11
>
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, PIG-2927-3.patch, PIG-2927-4.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476623#comment-13476623 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
Maven is on drugs. I try with stuff I had, no work. I pull from trunk, apply again, no work. Why can't it see the tar package?
compile-sources:
[javac] /Users/rjurney/Software/pig-trunk/build.xml:504: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 16 source files to /Users/rjurney/Software/pig-trunk/build/classes
[javac] /Users/rjurney/Software/pig-trunk/src/org/apache/pig/scripting/jruby/JrubyScriptEngine.java:51: package org.apache.pig.tar does not exist
[javac] import org.apache.pig.tar.TarUtils;
[javac] ^
[javac] /Users/rjurney/Software/pig-trunk/src/org/apache/pig/scripting/jruby/JrubyScriptEngine.java:170: cannot find symbol
[javac] symbol : variable TarUtils
[javac] location: class org.apache.pig.scripting.jruby.JrubyScriptEngine
[javac] TarUtils.tarFile(GEM_DIR_BASE_NAME, f.getParentFile(), new File(f, "lib"), os);
[javac] ^
[javac] Note: /Users/rjurney/Software/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 2 errors
BUILD FAILED
/Users/rjurney/Software/pig-trunk/build.xml:451: The following error occurred while executing this line:
/Users/rjurney/Software/pig-trunk/build.xml:504: Compile failed; see the compiler error output for details.
Total time: 14 seconds
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476545#comment-13476545 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
Thanks, patch applied. Testing now.
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Coveney updated PIG-2927:
----------------------------------
Attachment: PIG-2927-1.patch
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477452#comment-13477452 ]
Jonathan Coveney commented on PIG-2927:
---------------------------------------
Russell,
Make sure you try running stuff on the cluster. Otherwise it would have already worked, since it should pick up the gems from your environment.
As far as testing, that's tough. This is about shipping gems which you have and the server doesn't, but it is awkward to require that someone have gems in order to run the tests. I suppose we could download gems on the client, and then use them in M/R mode. Need to think on it beyond that.
Rohini, Cheolsoo,
I've added you as watchers to this not because I like annoying you, but since you guys have a lot more experience testing and keeping builds green than I do, I thought you might have some ideas.
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Fix For: 0.11
>
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462290#comment-13462290 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
Anything other than "Major", aka "never implement me."
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Reporter: Russell Jurney
> Priority: Minor
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Coveney updated PIG-2927:
----------------------------------
Attachment: PIG-2927-2.patch
Russell, thanks for your help on this :) I think it was picking up the dep from my environment. I think it should pull it from maven now.
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487252#comment-13487252 ]
Cheolsoo Park commented on PIG-2927:
------------------------------------
Hi Jon.
I am comfortable with your patch. Why don't we open a jira for adding an e2e test case for this feature and let this in 0.11?
Sounds reasonable? Let me know if anyone has any concerns.
Thanks!
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Fix For: 0.11
>
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Coveney reassigned PIG-2927:
-------------------------------------
Assignee: Jonathan Coveney (was: Russell Jurney)
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472118#comment-13472118 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
Thanks, will test this by next week.
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Attachments: PIG-2927-0.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477443#comment-13477443 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
Jon: Should I write e2e tests or unit tests? Not sure how to test this, can you advise? Don't mind doing the work.
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Fix For: 0.11
>
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468222#comment-13468222 ]
Jonathan Coveney commented on PIG-2927:
---------------------------------------
Oh, and of course, you need to have the gem (and dependencies) installed locally to wherever JRUBY_HOME is set.
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Attachments: PIG-2927-0.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2927:
-----------------------------------
Priority: Minor (was: Critical)
Assignee: (was: Jonathan Coveney)
Russel, "critical" is for critical bugs, not for desired new features. Example of something critical: Pig produces wrong data, Pig doesn't compile, Pig exhibits 10x performance regression.
For further elucidation, I refer you to Wikipedia: http://en.wikipedia.org/wiki/The_Boy_Who_Cried_Wolf
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Reporter: Russell Jurney
> Priority: Minor
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477442#comment-13477442 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
Initial test works:
---
register 'test.rb' using jruby as myfuncs;
A = load 'test/data/pigunit/top_queries_input_data.txt';
B = foreach A generate myfuncs.concat($0, $1);
---
require 'rubygems'
require 'pigudf'
require 'mail'
class Myudfs < PigUdf
def concat *input
input.inject(:+)
end
end
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Fix For: 0.11
>
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487486#comment-13487486 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
This sounds great!
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Fix For: 0.11
>
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481586#comment-13481586 ]
Jonathan Coveney commented on PIG-2927:
---------------------------------------
Thanks for taking a look, Cheolsoo!
A test would be awesome. IMHO we should try and use a non-GPL patch. For a test, there is no reason not to since we control all the variables. In the future if it is unavoidable, we can work around that.
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Fix For: 0.11
>
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478347#comment-13478347 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
Before JRuby gems even in local mode in UDFs did not actually work, and now they do, but that could be the effect of me setting $JRUBY_HOME. I'll test on the cluster.
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Fix For: 0.11
>
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464512#comment-13464512 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
I'm hearing that I can REGISTER these, as Jython does. I look into that.
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Russell Jurney
> Priority: Minor
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487240#comment-13487240 ]
Jonathan Coveney commented on PIG-2927:
---------------------------------------
Cheolsoo,
Was wondering what you think is necessary as far as getting this into 0.11. JRuby support is still evolving/experimental, so I think assuming this feature doesn't blow anything up (it doesn't), then the bar to putting it in doesn't need to be unduly high.
the e2e test you proposed seems like it'd be easy to do.
I need to update this patch to use the released version of 1.7.0, but besides that, would love thoughts.
Jon
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Fix For: 0.11
>
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Russell Jurney updated PIG-2927:
--------------------------------
Component/s: parser
Environment: JRuby UDFs
Affects Version/s: 0.11
Assignee: Russell Jurney
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Russell Jurney
> Priority: Minor
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Russell Jurney updated PIG-2927:
--------------------------------
Fix Version/s: 0.11
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Fix For: 0.11
>
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476559#comment-13476559 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
No build :(
[echo] *** Building Main Sources ***
[echo] *** To compile with all warnings enabled, supply -Dall.warnings=1 on command line ***
[echo] *** If all.warnings property is supplied, compile-sources-all-warnings target will be executed ***
[echo] *** Else, compile-sources (which only warns about deprecations) target will be executed ***
compile-sources:
[javac] /Users/rjurney/Software/pig-trunk/build.xml:503: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 14 source files to /Users/rjurney/Software/pig-trunk/build/classes
[javac] /Users/rjurney/Software/pig-trunk/src/org/apache/pig/scripting/jruby/JrubyScriptEngine.java:51: package org.apache.pig.tar does not exist
[javac] import org.apache.pig.tar.TarUtils;
[javac] ^
[javac] /Users/rjurney/Software/pig-trunk/src/org/apache/pig/scripting/jruby/JrubyScriptEngine.java:170: cannot find symbol
[javac] symbol : variable TarUtils
[javac] location: class org.apache.pig.scripting.jruby.JrubyScriptEngine
[javac] TarUtils.tarFile(GEM_DIR_BASE_NAME, f.getParentFile(), new File(f, "lib"), os);
[javac] ^
[javac] Note: /Users/rjurney/Software/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 2 errors
BUILD FAILED
/Users/rjurney/Software/pig-trunk/build.xml:451: The following error occurred while executing this line:
/Users/rjurney/Software/pig-trunk/build.xml:503: Compile failed; see the compiler error output for details.
Total time: 1 minute 0 seconds
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462392#comment-13462392 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
Moving from email.
http://stackoverflow.com/a/10083594/13969 shows you the installed path of a given gem.
jruby -S gem install mail
jirb
require 'rubygems'
spec = Gem::Specification.find_by_name("mail")
spec.gem_dir
But really... we can just use Bundler right? Its perfect - it expects a user to create their own gem environment. http://gembundler.com/ Then there is a list of gems in a jruby file, and we can go through that list and get the installed location in our bundler environment (which is likely to be in a subdirectory).
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Reporter: Russell Jurney
> Priority: Minor
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Coveney updated PIG-2927:
----------------------------------
Attachment: PIG-2927-3.patch
Wasn't maven. Was me overlooking the file in the git branch. Woops. Thanks for your patience, Russell.
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch, PIG-2927-3.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476471#comment-13476471 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
Testing now...
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Attachments: PIG-2927-0.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476494#comment-13476494 ]
Russell Jurney commented on PIG-2927:
-------------------------------------
Patch does not apply to trunk, please advise:
Russells-MacBook-Pro:pig-trunk rjurney$ patch -p0 <PIG-2927-0.patch
patching file build.xml
patching file ivy/libraries.properties
patching file src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
Reversed (or previously applied) patch detected! Assume -R? [n] y
Hunk #5 FAILED at 107.
Hunk #6 FAILED at 659.
2 out of 6 hunks FAILED -- saving rejects to file src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java.rej
patching file src/org/apache/pig/impl/util/JarManager.java
patching file src/org/apache/pig/scripting/jruby/JrubyAccumulatorEvalFunc.java
patching file src/org/apache/pig/scripting/jruby/JrubyAlgebraicEvalFunc.java
patching file src/org/apache/pig/scripting/jruby/JrubyEvalFunc.java
patching file src/org/apache/pig/scripting/jruby/JrubyScriptEngine.java
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Attachments: PIG-2927-0.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2927) SHIP and use JRuby gems in JRuby UDFs
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476540#comment-13476540 ]
Jonathan Coveney commented on PIG-2927:
---------------------------------------
Russell, I pulled trunk and made a new patch. Should apply fine now.
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on all machines on the cluster. Some way to SHIP them automatically with the job would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira