You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Min Zhou (JIRA)" <ji...@apache.org> on 2009/03/04 06:55:56 UTC
[jira] Created: (HIVE-322) cannot create temporary udf dynamically,
with a ClassNotFoundException
cannot create temporary udf dynamically, with a ClassNotFoundException
-----------------------------------------------------------------------
Key: HIVE-322
URL: https://issues.apache.org/jira/browse/HIVE-322
Project: Hadoop Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.3.0
Reporter: Min Zhou
Priority: Blocker
CREATE TEMPORARY FUNCTION strlen AS 'com.taobao.hadoop.hive.udf.UdfStringLength'
I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
scenario description:
I set a peroperty in hive-site.xml to configure the classpath of my udf.
<property>
<name>hive.aux.jars.path</name>
<value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
</property>
but failed to register it with a ClassNotFoundException when creating udf through the sql command.
I'll patch that soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-322) cannot create temporary udf
dynamically, with a ClassNotFoundException
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679902#action_12679902 ]
Joydeep Sen Sarma commented on HIVE-322:
----------------------------------------
there's already an 'add/delete/list <resource> ...' infrastructure on the cli. this should be part of that infrastructure (currently it only supports 'file' resource - but the idea was that it would support 'jar' and other kinds of resource in future as well).
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars-v3.patch, registerjars1.patch, registerjars2.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-322) cannot create temporary udf
dynamically, with a ClassNotFoundException
Posted by "Min Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678635#action_12678635 ]
Min Zhou commented on HIVE-322:
-------------------------------
I am considering adding a register statement for registering a jar to classpath. Agree?
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-322) cannot create temporary udf
dynamically, with a ClassNotFoundException
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma resolved HIVE-322.
------------------------------------
Resolution: Duplicate
duping to 322 that contains the fix
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars-v3.patch, registerjars1.patch, registerjars2.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HIVE-322) cannot create temporary udf
dynamically, with a ClassNotFoundException
Posted by "Min Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679206#action_12679206 ]
coderplay edited comment on HIVE-322 at 3/7/09 7:28 AM:
-------------------------------------------------------
The create function statement loads a function class staticly from $HIVE_AUX_JARS_PATH and hive.aux.jars.path
http://issues.apache.org/jira/browse/HADOOP-4070
We ofen start a Hive thrift server, and it will up for quite a time. We cannot dynamically load another UDF in another jar unless we restart the server .
It's not practical for us.
This patch introduces a register statement for appending a jar path to hive on-the-fly. It works well both in CLI and Thrift server mode.
usage:
hive>register '/home/zhoumin/zhoumin.jar';
hive>CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength';
hive>select strlen(time) from log_data;
Nevertheless, this patch cannot pass current unit test of hive, because the MapTask run during unit test will serialize MapRedWork w/o having done a register command. It will cause the ClassLoader cannot find my UDF. see line 61 and line 63 in MapTask.java.
was (Author: coderplay):
The create function statement loads a function class staticly from $HIVE_AUX_JARS_PATH and hive.aux.jars.path
http://issues.apache.org/jira/browse/HADOOP-4070
We ofen start a Hive thrift server, and it will up for quite a time. We cannot dynamically load another UDF in another jar unless we restart the server .
It's not practical for us.
This patch introduces a register statement for appending a jar path to hive on-the-fly. It works well both in CLI and Thrift server mode.
usage:
hive>register '/home/zhoumin/zhoumin.jar';
hive>CREATE TEMPORARY FUNCTION strlen AS 'taobao.hadoop.hive.udf.UdfStringLength';
hive>select strlen(time) from log_data;
Nevertheless, this patch cannot pass current unit test of hive, because the MapTask run during unit test will serialize MapRedWork w/o having done a register command. It will cause the ClassLoader cannot find my UDF. see line 61 and line 63 in MapTask.java.
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars-v3.patch, registerjars1.patch, registerjars2.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-322) cannot create temporary udf dynamically,
with a ClassNotFoundException
Posted by "Min Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Min Zhou updated HIVE-322:
--------------------------
Description:
I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
scenario description:
I set a peroperty in hive-site.xml to configure the classpath of my udf.
<property>
<name>hive.aux.jars.path</name>
<value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
</property>
but failed to register it with a ClassNotFoundException when creating udf through the sql command.
CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
I'll patch that soon.
was:
CREATE TEMPORARY FUNCTION strlen AS 'com.taobao.hadoop.hive.udf.UdfStringLength'
I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
scenario description:
I set a peroperty in hive-site.xml to configure the classpath of my udf.
<property>
<name>hive.aux.jars.path</name>
<value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
</property>
but failed to register it with a ClassNotFoundException when creating udf through the sql command.
I'll patch that soon.
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll patch that soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-322) cannot create temporary udf dynamically,
with a ClassNotFoundException
Posted by "Min Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Min Zhou updated HIVE-322:
--------------------------
Description:
I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
scenario description:
I set a peroperty in hive-site.xml to configure the classpath of my udf.
<property>
<name>hive.aux.jars.path</name>
<value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
</property>
but failed to register it with a ClassNotFoundException when creating udf through the sql command.
CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
I'll make a patch soon.
was:
I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
scenario description:
I set a peroperty in hive-site.xml to configure the classpath of my udf.
<property>
<name>hive.aux.jars.path</name>
<value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
</property>
but failed to register it with a ClassNotFoundException when creating udf through the sql command.
CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
I'll patch that soon.
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-322) cannot create temporary udf
dynamically, with a ClassNotFoundException
Posted by "Min Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682003#action_12682003 ]
Min Zhou commented on HIVE-322:
-------------------------------
Hi Joydeep,
pls check out this jira https://issues.apache.org/jira/browse/HIVE-338
It supports add file/jar on both cli & thriftserver now.
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars-v3.patch, registerjars1.patch, registerjars2.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-322) cannot create temporary udf
dynamically, with a ClassNotFoundException
Posted by "Min Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680131#action_12680131 ]
Min Zhou commented on HIVE-322:
-------------------------------
The add/delete/list <resource> ...' command is only supported by CLI mode right now. It puts a file path into a map, then be retrieved and added into hadoop's tmpfiles by ExecDriver's initialize method, but you know, this method won't be called by Thrift Server mode.
Have you considered adding this feature to Thrift Server mode?
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars-v3.patch, registerjars1.patch, registerjars2.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HIVE-322) cannot create temporary udf
dynamically, with a ClassNotFoundException
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682242#action_12682242 ]
Joydeep Sen Sarma edited comment on HIVE-322 at 3/16/09 1:04 AM:
-----------------------------------------------------------------
duping to 338 that contains the fix
was (Author: jsensarma):
duping to 322 that contains the fix
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars-v3.patch, registerjars1.patch, registerjars2.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HIVE-322) cannot create temporary udf
dynamically, with a ClassNotFoundException
Posted by "Min Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678720#action_12678720 ]
coderplay edited comment on HIVE-322 at 3/7/09 7:25 AM:
-------------------------------------------------------
it works, now.
hive> register '/home/zhoumin/myudf.jar' ;
OK
Time taken: 0.141 seconds
hive> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength';
OK
Time taken: 0.062 seconds
was (Author: coderplay):
it works, now.
hive> register '/home/zhoumin/myudf.jar' ;
OK
Time taken: 0.141 seconds
hive> CREATE TEMPORARY FUNCTION strlen AS 'com.taobao.hadoop.hive.udf.UdfStringLength';
OK
Time taken: 0.062 seconds
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars-v3.patch, registerjars1.patch, registerjars2.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-322) cannot create temporary udf dynamically,
with a ClassNotFoundException
Posted by "Min Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Min Zhou updated HIVE-322:
--------------------------
Attachment: registerjars-v3.patch
Can pass unit test now. but I need add another directory to store my udf jars, any better suggestions?
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars-v3.patch, registerjars1.patch, registerjars2.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-322) cannot create temporary udf dynamically,
with a ClassNotFoundException
Posted by "Min Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Min Zhou updated HIVE-322:
--------------------------
Attachment: registerjars2.patch
The create function statement loads a function class staticly from $HIVE_AUX_JARS_PATH and hive.aux.jars.path
http://issues.apache.org/jira/browse/HADOOP-4070
We ofen start a Hive thrift server, and it will up for quite a time. We cannot dynamically load another UDF in another jar unless we restart the server .
It's not practical for us.
This patch introduces a register statement for appending a jar path to hive on-the-fly. It works well both in CLI and Thrift server mode.
usage:
hive>register '/home/zhoumin/zhoumin.jar';
hive>CREATE TEMPORARY FUNCTION strlen AS 'taobao.hadoop.hive.udf.UdfStringLength';
hive>select strlen(time) from log_data;
Nevertheless, this patch cannot pass current unit test of hive, because the MapTask run during unit test will serialize MapRedWork w/o having done a register command. It will cause the ClassLoader cannot find my UDF. see line 61 and line 63 in MapTask.java.
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars1.patch, registerjars2.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-322) cannot create temporary udf dynamically,
with a ClassNotFoundException
Posted by "Min Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Min Zhou updated HIVE-322:
--------------------------
Attachment: registerjars1.patch
it works, now.
hive> register '/home/zhoumin/myudf.jar' ;
OK
Time taken: 0.141 seconds
hive> CREATE TEMPORARY FUNCTION strlen AS 'com.taobao.hadoop.hive.udf.UdfStringLength';
OK
Time taken: 0.062 seconds
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars1.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-322) cannot create temporary udf
dynamically, with a ClassNotFoundException
Posted by "Min Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680030#action_12680030 ]
Min Zhou commented on HIVE-322:
-------------------------------
Hey Joydeep, I'll try to implement it, however, we will make our UDFs temporarily on Thrift Server mode, not CLI.
That a great feature you mentioned!
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars-v3.patch, registerjars1.patch, registerjars2.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-322) cannot create temporary udf
dynamically, with a ClassNotFoundException
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680368#action_12680368 ]
Joydeep Sen Sarma commented on HIVE-322:
----------------------------------------
for the thrift server mode - where does the jar file sit? if it is available on the client side - doesn't it need to be transferred to the server side via the Thrift interface?
the set commands are also not available through the Thrift server - and i agree that all of these should be accessible via the Thrift interface (ignoring the file location issue). I think it should be fairly easy to refactor Thrift server and Driver/CliDriver code to make sure that all commands are supported on the ThriftServer (and don't think that we need to bring all the cli commands into the Hive grammar for that to happen).
let's open another jira for this issue (executing cli commands into thrift server) - it should be fairly easy to get in. if u can add the 'add jar' command to the cli in the meantime - we should be good to go. I looked at the patch and the main code looks good.
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars-v3.patch, registerjars1.patch, registerjars2.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-322) cannot create temporary udf
dynamically, with a ClassNotFoundException
Posted by "Min Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680726#action_12680726 ]
Min Zhou commented on HIVE-322:
-------------------------------
Files reside in tmpfiles, Jars in tmpjars of hadoop system.
My thrift server support add/delete file/jar, set, and normal HSQL query now , and the other commands I think is no need for thrift server.
I've created another jar, let's move there.
https://issues.apache.org/jira/browse/HIVE-338
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars-v3.patch, registerjars1.patch, registerjars2.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-322) cannot create temporary udf
dynamically, with a ClassNotFoundException
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681906#action_12681906 ]
Joydeep Sen Sarma commented on HIVE-322:
----------------------------------------
Hi Min - are you going to resubmit this as a 'add jar' cli command. should be very easy to do.
meanwhile i will review the thriftserver code.
> cannot create temporary udf dynamically, with a ClassNotFoundException
> -----------------------------------------------------------------------
>
> Key: HIVE-322
> URL: https://issues.apache.org/jira/browse/HIVE-322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0
> Reporter: Min Zhou
> Priority: Blocker
> Attachments: registerjars-v3.patch, registerjars1.patch, registerjars2.patch
>
>
> I found the ClassLoader cannot load my UDF when doing FunctionTask, because the ClassLoader hasnot append its classpaths on-the-fly yet.
> The ExecDriver' s addToClassPath(String[] newPaths) method is the only entry for ClassLoader dynamically append its classhpaths (besides hadoop's GenericOptionsParser).
> But that function wasnot called before FunctionTask getting my UDF class by class name. I think this is the reason why I came across that failure.
> scenario description:
> I set a peroperty in hive-site.xml to configure the classpath of my udf.
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/hadoop/hdpsoft/hive-auxs/zhoumin.jar</value>
> </property>
> but failed to register it with a ClassNotFoundException when creating udf through the sql command.
> CREATE TEMPORARY FUNCTION strlen AS 'hadoop.hive.udf.UdfStringLength'
> I'll make a patch soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.