You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org> on 2010/04/05 18:02:27 UTC
[jira] Created: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
UDFs for dynamic invocation of simple Java methods
--------------------------------------------------
Key: PIG-1354
URL: https://issues.apache.org/jira/browse/PIG-1354
Project: Pig
Issue Type: New Feature
Affects Versions: 0.8.0
Reporter: Dmitriy V. Ryaboy
Assignee: Dmitriy V. Ryaboy
Fix For: 0.8.0
The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-1354:
-----------------------------------
Status: Patch Available (was: Open)
Fixed audit release and findbugs warnings. I think.
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901577#action_12901577 ]
Olga Natkovich commented on PIG-1354:
-------------------------------------
Dmitry, Could you add release notes on how to use this?
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853421#action_12853421 ]
Dmitriy V. Ryaboy commented on PIG-1354:
----------------------------------------
The attached patch provides generic UDFs for invoking functions that return one of the following Java classes: String, Integer, Long, Double, Float, or their primitive equivalents. The same set of classes can serve as arguments to said functions.
Static or non-static methods can be invoked (a non-static method is assumed to be invoked on the first element in the argument list).
The method signature is supplied using Pig Latin's DEFINE statement.
Usage looks like this:
{code}
-- invoking a static method
DEFINE StringToLong InvokeForLong('java.lang.Long.valueOf', 'String')
longs = FOREACH strings GENERATE StringToLong(some_chararray);
-- invoking a method on an object
DEFINE StringConcat InvokeForString('java.lang.String.concat', 'String String', 'false')
concatenations = FOREACH strings GENERATE StringConcat(str1, str2);
{code}
There is some overhead to using this technique vs writing a method-specific UDF wrapper. In my tests, the dynamic version runs about 1.3x-1.8x slower than a non-dynamic version for simple methods.
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857943#action_12857943 ]
Dmitriy V. Ryaboy commented on PIG-1354:
----------------------------------------
Ashutosh, nice brainstorm, but I think that if you are to the point of actually writing java, you should write java. In a proper IDE. With highlighting, intellisense, javadoc lookup, and all those other awesome bells and whistles. Plus, reusable code and all that.
To Alan's earlier point -- I think it would be valuable to allow bags (transformed into arrays) as arguments. That opens up a large set of statistical functions from Apache Commons, for example. Wouldn't work for giant bags that don't fit into memory, but nice for smaller groups.
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-1354:
-----------------------------------
Attachment: PIG-1354.patch
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854246#action_12854246 ]
Alan Gates commented on PIG-1354:
---------------------------------
+1, looks good.
In my run of the tests it looks like the overhead of the reflection is about 3x. That's reasonable for quick and dirty work, and if people want better performance they can implement the method in a dedicated Pig Latin UDF.
One question, you have it set to do int, long, float, double, and string. Do you ever see a need to have byte[] <-> bytearray?
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853777#action_12853777 ]
Hadoop QA commented on PIG-1354:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12440782/PIG-1354.patch
against trunk revision 930168.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.
-1 release audit. The applied patch generated 524 release audit warnings (more than the trunk's current 523 warnings).
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/280/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/280/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/280/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/280/console
This message is automatically generated.
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857932#action_12857932 ]
Ashutosh Chauhan commented on PIG-1354:
---------------------------------------
Dmitriy,
Neat work!
This patch facilitates to call few existing methods in jdk libs(which are thus compiled and already available at runtime).
Thinking aloud, what will it take to go one step further from here. That is, to take uncompiled java code blocks and then compile them at runtime and make them available as udfs. If we can get that, then we can allow users to write java code of there udfs inline in pig script and then pig can compile it at runtime and do other necessary plumbing to make it work. And then, no more need to write java code separately, compile it, jar it, register it etc. All user code will be in one file. This will make writing udfs a lot easier.
http://docs.codehaus.org/display/JANINO/Home and http://commons.apache.org/jci/ might be of help.
Thoughts ?
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-1354:
-----------------------------------
Status: Open (was: Patch Available)
whaddaya know, findbugs found a bug.
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-1354:
-----------------------------------
Release Note: Please see PIG-1551 release notes.
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-1354:
-----------------------------------
Status: Patch Available (was: Open)
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-1354:
-----------------------------------
Attachment: PIG-1354.patch
fixed up the javadocs.
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853423#action_12853423 ]
Dmitriy V. Ryaboy commented on PIG-1354:
----------------------------------------
I should mention that this patch applies to Pig 0.6 and 0.7 as well.
I haven't tested with versions earlier than that, but theoretically there is no reason it wouldn't apply to all versions of Pig starting with 0.2.
If someone wants this functionality and doesn't want to wait for 0.8, just change the package from pig.builtin to piggybank or equivalent, and go generically forth.
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901584#action_12901584 ]
Dmitriy V. Ryaboy commented on PIG-1354:
----------------------------------------
Olga,
There is a follow-up ticket here: https://issues.apache.org/jira/browse/PIG-1551
If that gets committed, I have a pretty detailed explanation of how to use the stuff in http://squarecog.wordpress.com/2010/08/20/upcoming-features-in-pig-0-8-dynamic-invokers/ (happy to put the link in release notes, or just paste the whole post).
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854247#action_12854247 ]
Hadoop QA commented on PIG-1354:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12440935/PIG-1354.patch
against trunk revision 930168.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/281/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/281/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/281/console
This message is automatically generated.
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854260#action_12854260 ]
Dmitriy V. Ryaboy commented on PIG-1354:
----------------------------------------
Alan, I don't know that there are many methods that operate directly on byte arrays, but it would be easy to add if someone asks for it.
I will commit this (the test failure seems unrelated, see report below).
{code}
Error Message
Exception in constructor: testAggNoCombine (java.lang.ExceptionInInitializerError at org.apache.pig.test.TestBuiltin.<init>(TestBuiltin.java:65) at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at
java.lang.reflect.Constructor.newInstance(Constructor.java:513) at junit.framework.TestSuite.createTest(TestSuite.java:61) at
junit.framework.TestSuite.addTestMethod(TestSuite.java:283) at junit.framework.TestSuite.<init>(TestSuite.java:146) at
org.junit.internal.runners.JUnit38ClassRunner.<init>(JUnit38ClassRunner.java:67) at
org.junit.internal.builders.JUnit3Builder.runnerForClass(JUnit3Builder.java:14) at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57) at
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29) at
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57) at
org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24) at junit.framework.JUnit4TestAdapter.<init>(JUnit4TestAdapter.java:31) at
junit.framework.JUnit4TestAdapter.<init>(JUnit4TestAdapter.java:24) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at
java.lang.reflect.Constructor.newInstance(Constructor.java:513) at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:386) at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
Caused by: java.lang.IllegalArgumentException: port out of range:-1 at java.net.InetSocketAddress.<init>(InetSocketAddress.java:118) at
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:250) at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:202) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>
(NameNode.java:279) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956) at
org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:275) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:119)
at org.apache.pig.test.MiniCluster.setupMiniDfsAndMrClusters(MiniCluster.java:57) at org.apache.pig.test.MiniCluster.<init>(MiniCluster.java:47) at
org.apache.pig.test.MiniCluster.<clinit>(MiniCluster.java:44) ... 23 more )
Stacktrace
junit.framework.AssertionFailedError: Exception in constructor: testAggNoCombine (java.lang.ExceptionInInitializerError
at org.apache.pig.test.TestBuiltin.<init>(TestBuiltin.java:65)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
Caused by: java.lang.IllegalArgumentException: port out of range:-1
at java.net.InetSocketAddress.<init>(InetSocketAddress.java:118)
at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:250)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:202)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:275)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:119)
at org.apache.pig.test.MiniCluster.setupMiniDfsAndMrClusters(MiniCluster.java:57)
at org.apache.pig.test.MiniCluster.<init>(MiniCluster.java:47)
at org.apache.pig.test.MiniCluster.<clinit>(MiniCluster.java:44)
)
{code}
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-1354:
-----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed.
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-1354:
-----------------------------------
Attachment: PIG-1354.patch
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1354) UDFs for dynamic invocation of simple
Java methods
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901585#action_12901585 ]
Olga Natkovich commented on PIG-1354:
-------------------------------------
Sounds good, Dmitry. Richard will review and commit the patch and then please paste the release notes.
> UDFs for dynamic invocation of simple Java methods
> --------------------------------------------------
>
> Key: PIG-1354
> URL: https://issues.apache.org/jira/browse/PIG-1354
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.8.0
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1354.patch, PIG-1354.patch, PIG-1354.patch
>
>
> The need to create wrapper UDFs for simple Java functions creates unnecessary work for Pig users, slows down the development process, and produces a lot of trivial classes. We can use Java's reflection to allow invoking a number of methods on the fly, dynamically, by creating a generic UDF to accomplish this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.