You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Yakubovich, Alexey" <Al...@searshc.com> on 2015/01/02 21:00:50 UTC
Re: way to add custom udf jar in hadoop 2.x version
Try to look hr: http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
Another advice: insert your ADD JAR commands in your $HOME/.hiverc file and start hive. (http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@mail.gmail.com%3E)
From: Ted Yu <yu...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, December 31, 2014 at 8:25 AM
To: "dev@hive.apache.org<ma...@hive.apache.org>" <de...@hive.apache.org>>
Subject: Fwd: way to add custom udf jar in hadoop 2.x version
Forwarding Niels' question to hive mailing list.
On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <Ni...@basjes.nl>> wrote:
Thanks for the pointer.
This seems to work for functions. Is there something similar for CREATE EXTERNAL TABLE ??
Niels
On Dec 31, 2014 8:13 AM, "Ted Yu" <yu...@gmail.com>> wrote:
Have you seen this thread ?
http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jar&subj=Best+way+to+add+custom+UDF+jar+in+HiveServer2
On Dec 30, 2014, at 10:56 PM, reena upadhyay <re...@gmail.com>> wrote:
Hi,
I am using hadoop 2.4.0 version. I have created custom udf jar. I am trying to execute a simple select udf query using java hive jdbc client program. When hive execute the query using map reduce job, then the query execution get fails because the mapper is not able to locate the udf class.
So I wanted to add the udf jar in hadoop environment permanently. Please suggest me a way to add this external jar for single node and multi node hadoop cluster.
PS: I am using hive 0.13.1 version and I already have this custom udf jar added in HIVE_HOME/lib directory.
Thanks
This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.
Re: way to add custom udf jar in hadoop 2.x version
Posted by Niels Basjes <Ni...@basjes.nl>.
I created https://issues.apache.org/jira/browse/HIVE-9252 for this
improvement.
On Sun, Jan 4, 2015 at 5:16 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> Hi,
>
> These options:
> - HIVE_HOME/auxlib
> - http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
> - ADD JAR commands in your $HOME/.hiverc file
>
> either require IT operations to put my JAR on all nodes OR I cannot share
> it, Only works on the commandline and it won't work in HUE/Beeswax.
>
> Now "Permanent Functions":
> - https://issues.apache.org/jira/browse/HIVE-6047
> -
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions
>
> What these "Permanent Functions" do is:
> 1) put the jar on the cluster without IT operations putting the jar on all
> nodes
> 2) the jar is used transparently for everyone who want to use the function.
>
> I am writing a deserializer [1] (Not finished yet:
> https://github.com/nielsbasjes/logparser/blob/master/README-Hive.md) that
> should make existing files query-able as an external table in Hive.
>
> Question is: Is there something similar for CREATE EXTERNAL TABLE ??
>
> Something like
>
> CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
> ...
> STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
>
>
> Is this something for which there is already a JIRA (couldn't find it)?
> If not; Should I create one? (I.e. do you think this would make sense for
> others?)
>
> Niels Basjes
>
>
> On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey <
> Alexey.Yakubovich@searshc.com> wrote:
>
>> Try to look hr:
>> http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
>>
>> Another advice: insert your ADD JAR commands in your $HOME/.hiverc file
>> and start hive. (
>> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@mail.gmail.com%3E
>> )
>>
>>
>>
>> From: Ted Yu <yu...@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Date: Wednesday, December 31, 2014 at 8:25 AM
>> To: "dev@hive.apache.org" <de...@hive.apache.org>
>> Subject: Fwd: way to add custom udf jar in hadoop 2.x version
>>
>> Forwarding Niels' question to hive mailing list.
>>
>> On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>>
>>> Thanks for the pointer.
>>> This seems to work for functions. Is there something similar for CREATE
>>> EXTERNAL TABLE ??
>>>
>>> Niels
>>> On Dec 31, 2014 8:13 AM, "Ted Yu" <yu...@gmail.com> wrote:
>>>
>>>> Have you seen this thread ?
>>>>
>>>> http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jar&subj=Best+way+to+add+custom+UDF+jar+in+HiveServer2
>>>>
>>>> On Dec 30, 2014, at 10:56 PM, reena upadhyay <re...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am using hadoop 2.4.0 version. I have created custom udf jar. I am
>>>> trying to execute a simple select udf query using java hive jdbc client
>>>> program. When hive execute the query using map reduce job, then the query
>>>> execution get fails because the mapper is not able to locate the udf class.
>>>> So I wanted to add the udf jar in hadoop environment permanently.
>>>> Please suggest me a way to add this external jar for single node and multi
>>>> node hadoop cluster.
>>>>
>>>> PS: I am using hive 0.13.1 version and I already have this custom udf
>>>> jar added in HIVE_HOME/lib directory.
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>> This message, including any attachments, is the property of Sears
>> Holdings Corporation and/or one of its subsidiaries. It is confidential and
>> may contain proprietary or legally privileged information. If you are not
>> the intended recipient, please delete it without reading the contents.
>> Thank you.
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: way to add custom udf jar in hadoop 2.x version
Posted by Niels Basjes <Ni...@basjes.nl>.
I created https://issues.apache.org/jira/browse/HIVE-9252 for this
improvement.
On Sun, Jan 4, 2015 at 5:16 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> Hi,
>
> These options:
> - HIVE_HOME/auxlib
> - http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
> - ADD JAR commands in your $HOME/.hiverc file
>
> either require IT operations to put my JAR on all nodes OR I cannot share
> it, Only works on the commandline and it won't work in HUE/Beeswax.
>
> Now "Permanent Functions":
> - https://issues.apache.org/jira/browse/HIVE-6047
> -
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions
>
> What these "Permanent Functions" do is:
> 1) put the jar on the cluster without IT operations putting the jar on all
> nodes
> 2) the jar is used transparently for everyone who want to use the function.
>
> I am writing a deserializer [1] (Not finished yet:
> https://github.com/nielsbasjes/logparser/blob/master/README-Hive.md) that
> should make existing files query-able as an external table in Hive.
>
> Question is: Is there something similar for CREATE EXTERNAL TABLE ??
>
> Something like
>
> CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
> ...
> STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
>
>
> Is this something for which there is already a JIRA (couldn't find it)?
> If not; Should I create one? (I.e. do you think this would make sense for
> others?)
>
> Niels Basjes
>
>
> On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey <
> Alexey.Yakubovich@searshc.com> wrote:
>
>> Try to look hr:
>> http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
>>
>> Another advice: insert your ADD JAR commands in your $HOME/.hiverc file
>> and start hive. (
>> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@mail.gmail.com%3E
>> )
>>
>>
>>
>> From: Ted Yu <yu...@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Date: Wednesday, December 31, 2014 at 8:25 AM
>> To: "dev@hive.apache.org" <de...@hive.apache.org>
>> Subject: Fwd: way to add custom udf jar in hadoop 2.x version
>>
>> Forwarding Niels' question to hive mailing list.
>>
>> On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>>
>>> Thanks for the pointer.
>>> This seems to work for functions. Is there something similar for CREATE
>>> EXTERNAL TABLE ??
>>>
>>> Niels
>>> On Dec 31, 2014 8:13 AM, "Ted Yu" <yu...@gmail.com> wrote:
>>>
>>>> Have you seen this thread ?
>>>>
>>>> http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jar&subj=Best+way+to+add+custom+UDF+jar+in+HiveServer2
>>>>
>>>> On Dec 30, 2014, at 10:56 PM, reena upadhyay <re...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am using hadoop 2.4.0 version. I have created custom udf jar. I am
>>>> trying to execute a simple select udf query using java hive jdbc client
>>>> program. When hive execute the query using map reduce job, then the query
>>>> execution get fails because the mapper is not able to locate the udf class.
>>>> So I wanted to add the udf jar in hadoop environment permanently.
>>>> Please suggest me a way to add this external jar for single node and multi
>>>> node hadoop cluster.
>>>>
>>>> PS: I am using hive 0.13.1 version and I already have this custom udf
>>>> jar added in HIVE_HOME/lib directory.
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>> This message, including any attachments, is the property of Sears
>> Holdings Corporation and/or one of its subsidiaries. It is confidential and
>> may contain proprietary or legally privileged information. If you are not
>> the intended recipient, please delete it without reading the contents.
>> Thank you.
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: way to add custom udf jar in hadoop 2.x version
Posted by Niels Basjes <Ni...@basjes.nl>.
I created https://issues.apache.org/jira/browse/HIVE-9252 for this
improvement.
On Sun, Jan 4, 2015 at 5:16 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> Hi,
>
> These options:
> - HIVE_HOME/auxlib
> - http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
> - ADD JAR commands in your $HOME/.hiverc file
>
> either require IT operations to put my JAR on all nodes OR I cannot share
> it, Only works on the commandline and it won't work in HUE/Beeswax.
>
> Now "Permanent Functions":
> - https://issues.apache.org/jira/browse/HIVE-6047
> -
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions
>
> What these "Permanent Functions" do is:
> 1) put the jar on the cluster without IT operations putting the jar on all
> nodes
> 2) the jar is used transparently for everyone who want to use the function.
>
> I am writing a deserializer [1] (Not finished yet:
> https://github.com/nielsbasjes/logparser/blob/master/README-Hive.md) that
> should make existing files query-able as an external table in Hive.
>
> Question is: Is there something similar for CREATE EXTERNAL TABLE ??
>
> Something like
>
> CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
> ...
> STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
>
>
> Is this something for which there is already a JIRA (couldn't find it)?
> If not; Should I create one? (I.e. do you think this would make sense for
> others?)
>
> Niels Basjes
>
>
> On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey <
> Alexey.Yakubovich@searshc.com> wrote:
>
>> Try to look hr:
>> http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
>>
>> Another advice: insert your ADD JAR commands in your $HOME/.hiverc file
>> and start hive. (
>> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@mail.gmail.com%3E
>> )
>>
>>
>>
>> From: Ted Yu <yu...@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Date: Wednesday, December 31, 2014 at 8:25 AM
>> To: "dev@hive.apache.org" <de...@hive.apache.org>
>> Subject: Fwd: way to add custom udf jar in hadoop 2.x version
>>
>> Forwarding Niels' question to hive mailing list.
>>
>> On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>>
>>> Thanks for the pointer.
>>> This seems to work for functions. Is there something similar for CREATE
>>> EXTERNAL TABLE ??
>>>
>>> Niels
>>> On Dec 31, 2014 8:13 AM, "Ted Yu" <yu...@gmail.com> wrote:
>>>
>>>> Have you seen this thread ?
>>>>
>>>> http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jar&subj=Best+way+to+add+custom+UDF+jar+in+HiveServer2
>>>>
>>>> On Dec 30, 2014, at 10:56 PM, reena upadhyay <re...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am using hadoop 2.4.0 version. I have created custom udf jar. I am
>>>> trying to execute a simple select udf query using java hive jdbc client
>>>> program. When hive execute the query using map reduce job, then the query
>>>> execution get fails because the mapper is not able to locate the udf class.
>>>> So I wanted to add the udf jar in hadoop environment permanently.
>>>> Please suggest me a way to add this external jar for single node and multi
>>>> node hadoop cluster.
>>>>
>>>> PS: I am using hive 0.13.1 version and I already have this custom udf
>>>> jar added in HIVE_HOME/lib directory.
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>> This message, including any attachments, is the property of Sears
>> Holdings Corporation and/or one of its subsidiaries. It is confidential and
>> may contain proprietary or legally privileged information. If you are not
>> the intended recipient, please delete it without reading the contents.
>> Thank you.
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: way to add custom udf jar in hadoop 2.x version
Posted by Niels Basjes <Ni...@basjes.nl>.
I created https://issues.apache.org/jira/browse/HIVE-9252 for this
improvement.
On Sun, Jan 4, 2015 at 5:16 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> Hi,
>
> These options:
> - HIVE_HOME/auxlib
> - http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
> - ADD JAR commands in your $HOME/.hiverc file
>
> either require IT operations to put my JAR on all nodes OR I cannot share
> it, Only works on the commandline and it won't work in HUE/Beeswax.
>
> Now "Permanent Functions":
> - https://issues.apache.org/jira/browse/HIVE-6047
> -
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions
>
> What these "Permanent Functions" do is:
> 1) put the jar on the cluster without IT operations putting the jar on all
> nodes
> 2) the jar is used transparently for everyone who want to use the function.
>
> I am writing a deserializer [1] (Not finished yet:
> https://github.com/nielsbasjes/logparser/blob/master/README-Hive.md) that
> should make existing files query-able as an external table in Hive.
>
> Question is: Is there something similar for CREATE EXTERNAL TABLE ??
>
> Something like
>
> CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
> ...
> STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
>
>
> Is this something for which there is already a JIRA (couldn't find it)?
> If not; Should I create one? (I.e. do you think this would make sense for
> others?)
>
> Niels Basjes
>
>
> On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey <
> Alexey.Yakubovich@searshc.com> wrote:
>
>> Try to look hr:
>> http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
>>
>> Another advice: insert your ADD JAR commands in your $HOME/.hiverc file
>> and start hive. (
>> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@mail.gmail.com%3E
>> )
>>
>>
>>
>> From: Ted Yu <yu...@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Date: Wednesday, December 31, 2014 at 8:25 AM
>> To: "dev@hive.apache.org" <de...@hive.apache.org>
>> Subject: Fwd: way to add custom udf jar in hadoop 2.x version
>>
>> Forwarding Niels' question to hive mailing list.
>>
>> On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>>
>>> Thanks for the pointer.
>>> This seems to work for functions. Is there something similar for CREATE
>>> EXTERNAL TABLE ??
>>>
>>> Niels
>>> On Dec 31, 2014 8:13 AM, "Ted Yu" <yu...@gmail.com> wrote:
>>>
>>>> Have you seen this thread ?
>>>>
>>>> http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jar&subj=Best+way+to+add+custom+UDF+jar+in+HiveServer2
>>>>
>>>> On Dec 30, 2014, at 10:56 PM, reena upadhyay <re...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am using hadoop 2.4.0 version. I have created custom udf jar. I am
>>>> trying to execute a simple select udf query using java hive jdbc client
>>>> program. When hive execute the query using map reduce job, then the query
>>>> execution get fails because the mapper is not able to locate the udf class.
>>>> So I wanted to add the udf jar in hadoop environment permanently.
>>>> Please suggest me a way to add this external jar for single node and multi
>>>> node hadoop cluster.
>>>>
>>>> PS: I am using hive 0.13.1 version and I already have this custom udf
>>>> jar added in HIVE_HOME/lib directory.
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>> This message, including any attachments, is the property of Sears
>> Holdings Corporation and/or one of its subsidiaries. It is confidential and
>> may contain proprietary or legally privileged information. If you are not
>> the intended recipient, please delete it without reading the contents.
>> Thank you.
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: way to add custom udf jar in hadoop 2.x version
Posted by Niels Basjes <Ni...@basjes.nl>.
Hi,
These options:
- HIVE_HOME/auxlib
- http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
- ADD JAR commands in your $HOME/.hiverc file
either require IT operations to put my JAR on all nodes OR I cannot share
it, Only works on the commandline and it won't work in HUE/Beeswax.
Now "Permanent Functions":
- https://issues.apache.org/jira/browse/HIVE-6047
-
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions
What these "Permanent Functions" do is:
1) put the jar on the cluster without IT operations putting the jar on all
nodes
2) the jar is used transparently for everyone who want to use the function.
I am writing a deserializer [1] (Not finished yet:
https://github.com/nielsbasjes/logparser/blob/master/README-Hive.md) that
should make existing files query-able as an external table in Hive.
Question is: Is there something similar for CREATE EXTERNAL TABLE ??
Something like
CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
...
STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
[USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
Is this something for which there is already a JIRA (couldn't find it)?
If not; Should I create one? (I.e. do you think this would make sense for
others?)
Niels Basjes
On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey <
Alexey.Yakubovich@searshc.com> wrote:
> Try to look hr:
> http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
>
> Another advice: insert your ADD JAR commands in your $HOME/.hiverc file
> and start hive. (
> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@mail.gmail.com%3E
> )
>
>
>
> From: Ted Yu <yu...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Wednesday, December 31, 2014 at 8:25 AM
> To: "dev@hive.apache.org" <de...@hive.apache.org>
> Subject: Fwd: way to add custom udf jar in hadoop 2.x version
>
> Forwarding Niels' question to hive mailing list.
>
> On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>
>> Thanks for the pointer.
>> This seems to work for functions. Is there something similar for CREATE
>> EXTERNAL TABLE ??
>>
>> Niels
>> On Dec 31, 2014 8:13 AM, "Ted Yu" <yu...@gmail.com> wrote:
>>
>>> Have you seen this thread ?
>>>
>>> http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jar&subj=Best+way+to+add+custom+UDF+jar+in+HiveServer2
>>>
>>> On Dec 30, 2014, at 10:56 PM, reena upadhyay <re...@gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I am using hadoop 2.4.0 version. I have created custom udf jar. I am
>>> trying to execute a simple select udf query using java hive jdbc client
>>> program. When hive execute the query using map reduce job, then the query
>>> execution get fails because the mapper is not able to locate the udf class.
>>> So I wanted to add the udf jar in hadoop environment permanently. Please
>>> suggest me a way to add this external jar for single node and multi node
>>> hadoop cluster.
>>>
>>> PS: I am using hive 0.13.1 version and I already have this custom udf
>>> jar added in HIVE_HOME/lib directory.
>>>
>>>
>>> Thanks
>>>
>>>
> This message, including any attachments, is the property of Sears
> Holdings Corporation and/or one of its subsidiaries. It is confidential and
> may contain proprietary or legally privileged information. If you are not
> the intended recipient, please delete it without reading the contents.
> Thank you.
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: way to add custom udf jar in hadoop 2.x version
Posted by Niels Basjes <Ni...@basjes.nl>.
Hi,
These options:
- HIVE_HOME/auxlib
- http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
- ADD JAR commands in your $HOME/.hiverc file
either require IT operations to put my JAR on all nodes OR I cannot share
it, Only works on the commandline and it won't work in HUE/Beeswax.
Now "Permanent Functions":
- https://issues.apache.org/jira/browse/HIVE-6047
-
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions
What these "Permanent Functions" do is:
1) put the jar on the cluster without IT operations putting the jar on all
nodes
2) the jar is used transparently for everyone who want to use the function.
I am writing a deserializer [1] (Not finished yet:
https://github.com/nielsbasjes/logparser/blob/master/README-Hive.md) that
should make existing files query-able as an external table in Hive.
Question is: Is there something similar for CREATE EXTERNAL TABLE ??
Something like
CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
...
STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
[USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
Is this something for which there is already a JIRA (couldn't find it)?
If not; Should I create one? (I.e. do you think this would make sense for
others?)
Niels Basjes
On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey <
Alexey.Yakubovich@searshc.com> wrote:
> Try to look hr:
> http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
>
> Another advice: insert your ADD JAR commands in your $HOME/.hiverc file
> and start hive. (
> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@mail.gmail.com%3E
> )
>
>
>
> From: Ted Yu <yu...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Wednesday, December 31, 2014 at 8:25 AM
> To: "dev@hive.apache.org" <de...@hive.apache.org>
> Subject: Fwd: way to add custom udf jar in hadoop 2.x version
>
> Forwarding Niels' question to hive mailing list.
>
> On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>
>> Thanks for the pointer.
>> This seems to work for functions. Is there something similar for CREATE
>> EXTERNAL TABLE ??
>>
>> Niels
>> On Dec 31, 2014 8:13 AM, "Ted Yu" <yu...@gmail.com> wrote:
>>
>>> Have you seen this thread ?
>>>
>>> http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jar&subj=Best+way+to+add+custom+UDF+jar+in+HiveServer2
>>>
>>> On Dec 30, 2014, at 10:56 PM, reena upadhyay <re...@gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I am using hadoop 2.4.0 version. I have created custom udf jar. I am
>>> trying to execute a simple select udf query using java hive jdbc client
>>> program. When hive execute the query using map reduce job, then the query
>>> execution get fails because the mapper is not able to locate the udf class.
>>> So I wanted to add the udf jar in hadoop environment permanently. Please
>>> suggest me a way to add this external jar for single node and multi node
>>> hadoop cluster.
>>>
>>> PS: I am using hive 0.13.1 version and I already have this custom udf
>>> jar added in HIVE_HOME/lib directory.
>>>
>>>
>>> Thanks
>>>
>>>
> This message, including any attachments, is the property of Sears
> Holdings Corporation and/or one of its subsidiaries. It is confidential and
> may contain proprietary or legally privileged information. If you are not
> the intended recipient, please delete it without reading the contents.
> Thank you.
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: way to add custom udf jar in hadoop 2.x version
Posted by Niels Basjes <Ni...@basjes.nl>.
Hi,
These options:
- HIVE_HOME/auxlib
- http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
- ADD JAR commands in your $HOME/.hiverc file
either require IT operations to put my JAR on all nodes OR I cannot share
it, Only works on the commandline and it won't work in HUE/Beeswax.
Now "Permanent Functions":
- https://issues.apache.org/jira/browse/HIVE-6047
-
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions
What these "Permanent Functions" do is:
1) put the jar on the cluster without IT operations putting the jar on all
nodes
2) the jar is used transparently for everyone who want to use the function.
I am writing a deserializer [1] (Not finished yet:
https://github.com/nielsbasjes/logparser/blob/master/README-Hive.md) that
should make existing files query-able as an external table in Hive.
Question is: Is there something similar for CREATE EXTERNAL TABLE ??
Something like
CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
...
STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
[USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
Is this something for which there is already a JIRA (couldn't find it)?
If not; Should I create one? (I.e. do you think this would make sense for
others?)
Niels Basjes
On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey <
Alexey.Yakubovich@searshc.com> wrote:
> Try to look hr:
> http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
>
> Another advice: insert your ADD JAR commands in your $HOME/.hiverc file
> and start hive. (
> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@mail.gmail.com%3E
> )
>
>
>
> From: Ted Yu <yu...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Wednesday, December 31, 2014 at 8:25 AM
> To: "dev@hive.apache.org" <de...@hive.apache.org>
> Subject: Fwd: way to add custom udf jar in hadoop 2.x version
>
> Forwarding Niels' question to hive mailing list.
>
> On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>
>> Thanks for the pointer.
>> This seems to work for functions. Is there something similar for CREATE
>> EXTERNAL TABLE ??
>>
>> Niels
>> On Dec 31, 2014 8:13 AM, "Ted Yu" <yu...@gmail.com> wrote:
>>
>>> Have you seen this thread ?
>>>
>>> http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jar&subj=Best+way+to+add+custom+UDF+jar+in+HiveServer2
>>>
>>> On Dec 30, 2014, at 10:56 PM, reena upadhyay <re...@gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I am using hadoop 2.4.0 version. I have created custom udf jar. I am
>>> trying to execute a simple select udf query using java hive jdbc client
>>> program. When hive execute the query using map reduce job, then the query
>>> execution get fails because the mapper is not able to locate the udf class.
>>> So I wanted to add the udf jar in hadoop environment permanently. Please
>>> suggest me a way to add this external jar for single node and multi node
>>> hadoop cluster.
>>>
>>> PS: I am using hive 0.13.1 version and I already have this custom udf
>>> jar added in HIVE_HOME/lib directory.
>>>
>>>
>>> Thanks
>>>
>>>
> This message, including any attachments, is the property of Sears
> Holdings Corporation and/or one of its subsidiaries. It is confidential and
> may contain proprietary or legally privileged information. If you are not
> the intended recipient, please delete it without reading the contents.
> Thank you.
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: way to add custom udf jar in hadoop 2.x version
Posted by Niels Basjes <Ni...@basjes.nl>.
Hi,
These options:
- HIVE_HOME/auxlib
- http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
- ADD JAR commands in your $HOME/.hiverc file
either require IT operations to put my JAR on all nodes OR I cannot share
it, Only works on the commandline and it won't work in HUE/Beeswax.
Now "Permanent Functions":
- https://issues.apache.org/jira/browse/HIVE-6047
-
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions
What these "Permanent Functions" do is:
1) put the jar on the cluster without IT operations putting the jar on all
nodes
2) the jar is used transparently for everyone who want to use the function.
I am writing a deserializer [1] (Not finished yet:
https://github.com/nielsbasjes/logparser/blob/master/README-Hive.md) that
should make existing files query-able as an external table in Hive.
Question is: Is there something similar for CREATE EXTERNAL TABLE ??
Something like
CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
...
STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
[USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
Is this something for which there is already a JIRA (couldn't find it)?
If not; Should I create one? (I.e. do you think this would make sense for
others?)
Niels Basjes
On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey <
Alexey.Yakubovich@searshc.com> wrote:
> Try to look hr:
> http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
>
> Another advice: insert your ADD JAR commands in your $HOME/.hiverc file
> and start hive. (
> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@mail.gmail.com%3E
> )
>
>
>
> From: Ted Yu <yu...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Wednesday, December 31, 2014 at 8:25 AM
> To: "dev@hive.apache.org" <de...@hive.apache.org>
> Subject: Fwd: way to add custom udf jar in hadoop 2.x version
>
> Forwarding Niels' question to hive mailing list.
>
> On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>
>> Thanks for the pointer.
>> This seems to work for functions. Is there something similar for CREATE
>> EXTERNAL TABLE ??
>>
>> Niels
>> On Dec 31, 2014 8:13 AM, "Ted Yu" <yu...@gmail.com> wrote:
>>
>>> Have you seen this thread ?
>>>
>>> http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jar&subj=Best+way+to+add+custom+UDF+jar+in+HiveServer2
>>>
>>> On Dec 30, 2014, at 10:56 PM, reena upadhyay <re...@gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I am using hadoop 2.4.0 version. I have created custom udf jar. I am
>>> trying to execute a simple select udf query using java hive jdbc client
>>> program. When hive execute the query using map reduce job, then the query
>>> execution get fails because the mapper is not able to locate the udf class.
>>> So I wanted to add the udf jar in hadoop environment permanently. Please
>>> suggest me a way to add this external jar for single node and multi node
>>> hadoop cluster.
>>>
>>> PS: I am using hive 0.13.1 version and I already have this custom udf
>>> jar added in HIVE_HOME/lib directory.
>>>
>>>
>>> Thanks
>>>
>>>
> This message, including any attachments, is the property of Sears
> Holdings Corporation and/or one of its subsidiaries. It is confidential and
> may contain proprietary or legally privileged information. If you are not
> the intended recipient, please delete it without reading the contents.
> Thank you.
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes