You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2010/07/02 01:35:49 UTC
[jira] Created: (HIVE-1447) Speed up reflection method calls
Speed up reflection method calls
--------------------------------
Key: HIVE-1447
URL: https://issues.apache.org/jira/browse/HIVE-1447
Project: Hadoop Hive
Issue Type: Improvement
Reporter: Zheng Shao
Assignee: Zheng Shao
See http://www.cowtowncoder.com/blog/archives/2010/04/entry_396.html and http://www.jguru.com/faq/view.jsp?EID=246569
There is a huge drop of overhead (more than half) if we do "field.setAccessible(true)" for the field that we want to access.
I did a simple experiment and that worked well with method as well.
The results are (note that the method just add 1 to an integer):
{code}
100000000 regular method calls:26 milliseconds.
100000000 reflective method calls without lookup:4029 milliseconds.
100000000 accessible reflective method calls without lookup:1810 milliseconds.
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1447) Speed up reflection method calls
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884488#action_12884488 ]
Edward Capriolo commented on HIVE-1447:
---------------------------------------
Does this have an affect on all generic UDFs? I had someone telling me that they added one of my genUDF's to a query on a large table and the time to complete the job went up dramatically.
> Speed up reflection method calls
> --------------------------------
>
> Key: HIVE-1447
> URL: https://issues.apache.org/jira/browse/HIVE-1447
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: A.java, HIVE-1447.1.patch
>
>
> See http://www.cowtowncoder.com/blog/archives/2010/04/entry_396.html and http://www.jguru.com/faq/view.jsp?EID=246569
> There is a huge drop of overhead (more than half) if we do "field.setAccessible(true)" for the field that we want to access.
> I did a simple experiment and that worked well with method as well.
> The results are (note that the method just add 1 to an integer):
> {code}
> 100000000 regular method calls:26 milliseconds.
> 100000000 reflective method calls without lookup:4029 milliseconds.
> 100000000 accessible reflective method calls without lookup:1810 milliseconds.
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1447) Speed up reflection method calls
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884491#action_12884491 ]
Zheng Shao commented on HIVE-1447:
----------------------------------
GenericUDF should not be affected.
GenericUDFBridge is a bridge betwen GenericUDF and UDF. This patch will fix all problems for UDF and UDAF.
> Speed up reflection method calls
> --------------------------------
>
> Key: HIVE-1447
> URL: https://issues.apache.org/jira/browse/HIVE-1447
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: A.java, HIVE-1447.1.patch
>
>
> See http://www.cowtowncoder.com/blog/archives/2010/04/entry_396.html and http://www.jguru.com/faq/view.jsp?EID=246569
> There is a huge drop of overhead (more than half) if we do "field.setAccessible(true)" for the field that we want to access.
> I did a simple experiment and that worked well with method as well.
> The results are (note that the method just add 1 to an integer):
> {code}
> 100000000 regular method calls:26 milliseconds.
> 100000000 reflective method calls without lookup:4029 milliseconds.
> 100000000 accessible reflective method calls without lookup:1810 milliseconds.
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1447) Speed up reflection method calls
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang updated HIVE-1447:
-------------------------------
Status: Resolved (was: Patch Available)
Fix Version/s: 0.7.0
Resolution: Fixed
Committed. Thanks Zheng!
> Speed up reflection method calls
> --------------------------------
>
> Key: HIVE-1447
> URL: https://issues.apache.org/jira/browse/HIVE-1447
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.7.0
>
> Attachments: A.java, HIVE-1447.1.patch
>
>
> See http://www.cowtowncoder.com/blog/archives/2010/04/entry_396.html and http://www.jguru.com/faq/view.jsp?EID=246569
> There is a huge drop of overhead (more than half) if we do "field.setAccessible(true)" for the field that we want to access.
> I did a simple experiment and that worked well with method as well.
> The results are (note that the method just add 1 to an integer):
> {code}
> 100000000 regular method calls:26 milliseconds.
> 100000000 reflective method calls without lookup:4029 milliseconds.
> 100000000 accessible reflective method calls without lookup:1810 milliseconds.
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1447) Speed up reflection method calls
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-1447:
-----------------------------
Status: Patch Available (was: Open)
> Speed up reflection method calls
> --------------------------------
>
> Key: HIVE-1447
> URL: https://issues.apache.org/jira/browse/HIVE-1447
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: A.java, HIVE-1447.1.patch
>
>
> See http://www.cowtowncoder.com/blog/archives/2010/04/entry_396.html and http://www.jguru.com/faq/view.jsp?EID=246569
> There is a huge drop of overhead (more than half) if we do "field.setAccessible(true)" for the field that we want to access.
> I did a simple experiment and that worked well with method as well.
> The results are (note that the method just add 1 to an integer):
> {code}
> 100000000 regular method calls:26 milliseconds.
> 100000000 reflective method calls without lookup:4029 milliseconds.
> 100000000 accessible reflective method calls without lookup:1810 milliseconds.
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1447) Speed up reflection method calls
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-1447:
-----------------------------
Attachment: HIVE-1447.1.patch
Patch that sets "setAccessible" for both GenericUDFBridge.java and GenericUDAFBridge.java
> Speed up reflection method calls
> --------------------------------
>
> Key: HIVE-1447
> URL: https://issues.apache.org/jira/browse/HIVE-1447
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: A.java, HIVE-1447.1.patch
>
>
> See http://www.cowtowncoder.com/blog/archives/2010/04/entry_396.html and http://www.jguru.com/faq/view.jsp?EID=246569
> There is a huge drop of overhead (more than half) if we do "field.setAccessible(true)" for the field that we want to access.
> I did a simple experiment and that worked well with method as well.
> The results are (note that the method just add 1 to an integer):
> {code}
> 100000000 regular method calls:26 milliseconds.
> 100000000 reflective method calls without lookup:4029 milliseconds.
> 100000000 accessible reflective method calls without lookup:1810 milliseconds.
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1447) Speed up reflection method calls
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-1447:
-----------------------------
Attachment: A.java
A.java for performance test. Some of the code is borrowed from http://www.jguru.com/faq/view.jsp?EID=246569
> Speed up reflection method calls
> --------------------------------
>
> Key: HIVE-1447
> URL: https://issues.apache.org/jira/browse/HIVE-1447
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: A.java
>
>
> See http://www.cowtowncoder.com/blog/archives/2010/04/entry_396.html and http://www.jguru.com/faq/view.jsp?EID=246569
> There is a huge drop of overhead (more than half) if we do "field.setAccessible(true)" for the field that we want to access.
> I did a simple experiment and that worked well with method as well.
> The results are (note that the method just add 1 to an integer):
> {code}
> 100000000 regular method calls:26 milliseconds.
> 100000000 reflective method calls without lookup:4029 milliseconds.
> 100000000 accessible reflective method calls without lookup:1810 milliseconds.
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1447) Speed up reflection method calls
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884605#action_12884605 ]
He Yongqiang commented on HIVE-1447:
------------------------------------
will commit once tests pass
> Speed up reflection method calls
> --------------------------------
>
> Key: HIVE-1447
> URL: https://issues.apache.org/jira/browse/HIVE-1447
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: A.java, HIVE-1447.1.patch
>
>
> See http://www.cowtowncoder.com/blog/archives/2010/04/entry_396.html and http://www.jguru.com/faq/view.jsp?EID=246569
> There is a huge drop of overhead (more than half) if we do "field.setAccessible(true)" for the field that we want to access.
> I did a simple experiment and that worked well with method as well.
> The results are (note that the method just add 1 to an integer):
> {code}
> 100000000 regular method calls:26 milliseconds.
> 100000000 reflective method calls without lookup:4029 milliseconds.
> 100000000 accessible reflective method calls without lookup:1810 milliseconds.
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.