You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Erwan Queffélec <er...@gmail.com> on 2015/06/29 14:30:07 UTC
Hive and elasticsearch-hadoop-2.1.0 : NoClassDefFoundError
Hello,
I'm running HortonWorks HDP 2.2.6, hive 0.14 alongside an ElasticSearch
cluster
For some reason Hive can't seem to connect to my ES cluster Using the ES
SerDe (I'm using elasticsearch-hadoop-2.1.0.jar). When time comes for my
job to insert the my query result in an ES resources, I get this stacktrace:
NoClassDefFoundError: org/apache/commons/httpclient/URIException
at
org.elasticsearch.hadoop.hive.HiveUtils.structObjectInspector(HiveUtils.java:57)
at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:82)
at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:97)
I'm quite puzzled because commons-httpclient is supposed to be on the
hive-client classpath :
# ls -l /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
-rw-r--r-- 1 root root 279781 Mar 31 20:26
/usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
# ls -l /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
-rw-r--r-- 1 root root 279781 Mar 31 20:26
/usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
What am I missing ?
Thanks a lot for your help,
Kind regards,
Erwan
Re: Hive and elasticsearch-hadoop-2.1.0 : NoClassDefFoundError
Posted by Nitin Pawar <ni...@gmail.com>.
great it helped
On Mon, Jun 29, 2015 at 7:29 PM, Erwan Queffélec <er...@gmail.com>
wrote:
> [continued]
> A dependency for a custom UDF seems not to be properly shaded, as I could
> see in an excerpt of the maven build output:
> [INFO] Including org.apache.httpcomponents:httpclient:jar:4.1.2 in the
> shaded jar.
> [INFO] Including org.apache.httpcomponents:httpcore:jar:4.1.2 in the
> shaded jar.
>
> I'm going to look into this. Thanks a lot for confirming things worked as
> I expected on your end!
>
> Regards,
>
> Erwan
>
> On Mon, Jun 29, 2015 at 3:55 PM, Erwan Queffélec <
> erwan.queffelec@gmail.com> wrote:
>
>> Hi Nitin,
>>
>> Digging up a bit I discovered that the error is probably on our end :
>>
>>
>>
>> On Mon, Jun 29, 2015 at 3:54 PM, Nitin Pawar <ni...@gmail.com>
>> wrote:
>>
>>> I am using 2.2.4-2.2 but did not get any error.
>>>
>>> can you check what all services are installed on the node where hive
>>> client is running
>>>
>>> On Mon, Jun 29, 2015 at 7:18 PM, Erwan Queffélec <
>>> erwan.queffelec@gmail.com> wrote:
>>>
>>>> Hi Nitin,
>>>>
>>>> No, I didn't do such a thing. I'm using the stock 0.14 version from HDP
>>>> 2.2.4 (said 2.2.6 earlier but that was wrong)
>>>>
>>>> # hive --version
>>>> Hive 0.14.0.2.2.4.2-2
>>>> Subversion
>>>> git://ip-10-0-0-5.ec2.internal/grid/0/jenkins/workspace/HDP-2.2.4.1-centos6/bigtop/build/hive/rpm/BUILD/hive-0.14.0.2.2.4.2
>>>> -r 115d99896f5a4a81e7d91e052e8d38d7436b78d4
>>>> Compiled by jenkins on Tue Mar 31 16:26:33 EDT 2015
>>>> From source with checksum 1f34a1d4e566c3e801582862ed85ee93
>>>>
>>>> Thanks for taking the time.
>>>>
>>>> Kind regards,
>>>>
>>>> Erwan
>>>>
>>>> On Mon, Jun 29, 2015 at 3:44 PM, Nitin Pawar <ni...@gmail.com>
>>>> wrote:
>>>>
>>>>> by any chance you built hive yourself ?
>>>>>
>>>>> On Mon, Jun 29, 2015 at 7:11 PM, Erwan Queffélec <
>>>>> erwan.queffelec@gmail.com> wrote:
>>>>>
>>>>>> Additional info : it works when I manually add the jar with ADD JAR
>>>>>> <file>;
>>>>>>
>>>>>> hive> ADD JAR '
>>>>>> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar'
>>>>>>
>>>>>> I'm quite new to hive and hadoop in general. This is my first post on
>>>>>> this mailing list, so please excuse me if the folowing question has been
>>>>>> asked & answered over and over again :
>>>>>>
>>>>>> Perhaps I'm a bit naíve, but I though that hive Custom SerDe/UD*Fs
>>>>>> were able to access everything already on the Hive classpath. Was it just a
>>>>>> dream ?
>>>>>>
>>>>>> I would greatly appreciate some pointers, thanks to anyone who might
>>>>>> be able to help !
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Erwan
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 29, 2015 at 2:30 PM, Erwan Queffélec <
>>>>>> erwan.queffelec@gmail.com> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm running HortonWorks HDP 2.2.6, hive 0.14 alongside an
>>>>>>> ElasticSearch cluster
>>>>>>>
>>>>>>> For some reason Hive can't seem to connect to my ES cluster Using
>>>>>>> the ES SerDe (I'm using elasticsearch-hadoop-2.1.0.jar). When time comes
>>>>>>> for my job to insert the my query result in an ES resources, I get this
>>>>>>> stacktrace:
>>>>>>>
>>>>>>> NoClassDefFoundError: org/apache/commons/httpclient/URIException
>>>>>>> at
>>>>>>> org.elasticsearch.hadoop.hive.HiveUtils.structObjectInspector(HiveUtils.java:57)
>>>>>>> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:82)
>>>>>>> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:97)
>>>>>>>
>>>>>>> I'm quite puzzled because commons-httpclient is supposed to be on
>>>>>>> the hive-client classpath :
>>>>>>>
>>>>>>> # ls -l
>>>>>>> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
>>>>>>> -rw-r--r-- 1 root root 279781 Mar 31 20:26
>>>>>>> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
>>>>>>> # ls -l /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
>>>>>>> -rw-r--r-- 1 root root 279781 Mar 31 20:26
>>>>>>> /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
>>>>>>>
>>>>>>> What am I missing ?
>>>>>>>
>>>>>>> Thanks a lot for your help,
>>>>>>>
>>>>>>> Kind regards,
>>>>>>>
>>>>>>> Erwan
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Nitin Pawar
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>
--
Nitin Pawar
Re: Hive and elasticsearch-hadoop-2.1.0 : NoClassDefFoundError
Posted by Erwan Queffélec <er...@gmail.com>.
[continued]
A dependency for a custom UDF seems not to be properly shaded, as I could
see in an excerpt of the maven build output:
[INFO] Including org.apache.httpcomponents:httpclient:jar:4.1.2 in the
shaded jar.
[INFO] Including org.apache.httpcomponents:httpcore:jar:4.1.2 in the shaded
jar.
I'm going to look into this. Thanks a lot for confirming things worked as I
expected on your end!
Regards,
Erwan
On Mon, Jun 29, 2015 at 3:55 PM, Erwan Queffélec <er...@gmail.com>
wrote:
> Hi Nitin,
>
> Digging up a bit I discovered that the error is probably on our end :
>
>
>
> On Mon, Jun 29, 2015 at 3:54 PM, Nitin Pawar <ni...@gmail.com>
> wrote:
>
>> I am using 2.2.4-2.2 but did not get any error.
>>
>> can you check what all services are installed on the node where hive
>> client is running
>>
>> On Mon, Jun 29, 2015 at 7:18 PM, Erwan Queffélec <
>> erwan.queffelec@gmail.com> wrote:
>>
>>> Hi Nitin,
>>>
>>> No, I didn't do such a thing. I'm using the stock 0.14 version from HDP
>>> 2.2.4 (said 2.2.6 earlier but that was wrong)
>>>
>>> # hive --version
>>> Hive 0.14.0.2.2.4.2-2
>>> Subversion
>>> git://ip-10-0-0-5.ec2.internal/grid/0/jenkins/workspace/HDP-2.2.4.1-centos6/bigtop/build/hive/rpm/BUILD/hive-0.14.0.2.2.4.2
>>> -r 115d99896f5a4a81e7d91e052e8d38d7436b78d4
>>> Compiled by jenkins on Tue Mar 31 16:26:33 EDT 2015
>>> From source with checksum 1f34a1d4e566c3e801582862ed85ee93
>>>
>>> Thanks for taking the time.
>>>
>>> Kind regards,
>>>
>>> Erwan
>>>
>>> On Mon, Jun 29, 2015 at 3:44 PM, Nitin Pawar <ni...@gmail.com>
>>> wrote:
>>>
>>>> by any chance you built hive yourself ?
>>>>
>>>> On Mon, Jun 29, 2015 at 7:11 PM, Erwan Queffélec <
>>>> erwan.queffelec@gmail.com> wrote:
>>>>
>>>>> Additional info : it works when I manually add the jar with ADD JAR
>>>>> <file>;
>>>>>
>>>>> hive> ADD JAR '
>>>>> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar'
>>>>>
>>>>> I'm quite new to hive and hadoop in general. This is my first post on
>>>>> this mailing list, so please excuse me if the folowing question has been
>>>>> asked & answered over and over again :
>>>>>
>>>>> Perhaps I'm a bit naíve, but I though that hive Custom SerDe/UD*Fs
>>>>> were able to access everything already on the Hive classpath. Was it just a
>>>>> dream ?
>>>>>
>>>>> I would greatly appreciate some pointers, thanks to anyone who might
>>>>> be able to help !
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Erwan
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jun 29, 2015 at 2:30 PM, Erwan Queffélec <
>>>>> erwan.queffelec@gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm running HortonWorks HDP 2.2.6, hive 0.14 alongside an
>>>>>> ElasticSearch cluster
>>>>>>
>>>>>> For some reason Hive can't seem to connect to my ES cluster Using the
>>>>>> ES SerDe (I'm using elasticsearch-hadoop-2.1.0.jar). When time comes for my
>>>>>> job to insert the my query result in an ES resources, I get this stacktrace:
>>>>>>
>>>>>> NoClassDefFoundError: org/apache/commons/httpclient/URIException
>>>>>> at
>>>>>> org.elasticsearch.hadoop.hive.HiveUtils.structObjectInspector(HiveUtils.java:57)
>>>>>> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:82)
>>>>>> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:97)
>>>>>>
>>>>>> I'm quite puzzled because commons-httpclient is supposed to be on the
>>>>>> hive-client classpath :
>>>>>>
>>>>>> # ls -l /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
>>>>>> -rw-r--r-- 1 root root 279781 Mar 31 20:26
>>>>>> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
>>>>>> # ls -l /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
>>>>>> -rw-r--r-- 1 root root 279781 Mar 31 20:26
>>>>>> /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
>>>>>>
>>>>>> What am I missing ?
>>>>>>
>>>>>> Thanks a lot for your help,
>>>>>>
>>>>>> Kind regards,
>>>>>>
>>>>>> Erwan
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Nitin Pawar
>>>>
>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
Re: Hive and elasticsearch-hadoop-2.1.0 : NoClassDefFoundError
Posted by Erwan Queffélec <er...@gmail.com>.
Hi Nitin,
Digging up a bit I discovered that the error is probably on our end :
On Mon, Jun 29, 2015 at 3:54 PM, Nitin Pawar <ni...@gmail.com>
wrote:
> I am using 2.2.4-2.2 but did not get any error.
>
> can you check what all services are installed on the node where hive
> client is running
>
> On Mon, Jun 29, 2015 at 7:18 PM, Erwan Queffélec <
> erwan.queffelec@gmail.com> wrote:
>
>> Hi Nitin,
>>
>> No, I didn't do such a thing. I'm using the stock 0.14 version from HDP
>> 2.2.4 (said 2.2.6 earlier but that was wrong)
>>
>> # hive --version
>> Hive 0.14.0.2.2.4.2-2
>> Subversion
>> git://ip-10-0-0-5.ec2.internal/grid/0/jenkins/workspace/HDP-2.2.4.1-centos6/bigtop/build/hive/rpm/BUILD/hive-0.14.0.2.2.4.2
>> -r 115d99896f5a4a81e7d91e052e8d38d7436b78d4
>> Compiled by jenkins on Tue Mar 31 16:26:33 EDT 2015
>> From source with checksum 1f34a1d4e566c3e801582862ed85ee93
>>
>> Thanks for taking the time.
>>
>> Kind regards,
>>
>> Erwan
>>
>> On Mon, Jun 29, 2015 at 3:44 PM, Nitin Pawar <ni...@gmail.com>
>> wrote:
>>
>>> by any chance you built hive yourself ?
>>>
>>> On Mon, Jun 29, 2015 at 7:11 PM, Erwan Queffélec <
>>> erwan.queffelec@gmail.com> wrote:
>>>
>>>> Additional info : it works when I manually add the jar with ADD JAR
>>>> <file>;
>>>>
>>>> hive> ADD JAR '
>>>> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar'
>>>>
>>>> I'm quite new to hive and hadoop in general. This is my first post on
>>>> this mailing list, so please excuse me if the folowing question has been
>>>> asked & answered over and over again :
>>>>
>>>> Perhaps I'm a bit naíve, but I though that hive Custom SerDe/UD*Fs were
>>>> able to access everything already on the Hive classpath. Was it just a
>>>> dream ?
>>>>
>>>> I would greatly appreciate some pointers, thanks to anyone who might be
>>>> able to help !
>>>>
>>>> Best regards,
>>>>
>>>> Erwan
>>>>
>>>>
>>>>
>>>> On Mon, Jun 29, 2015 at 2:30 PM, Erwan Queffélec <
>>>> erwan.queffelec@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm running HortonWorks HDP 2.2.6, hive 0.14 alongside an
>>>>> ElasticSearch cluster
>>>>>
>>>>> For some reason Hive can't seem to connect to my ES cluster Using the
>>>>> ES SerDe (I'm using elasticsearch-hadoop-2.1.0.jar). When time comes for my
>>>>> job to insert the my query result in an ES resources, I get this stacktrace:
>>>>>
>>>>> NoClassDefFoundError: org/apache/commons/httpclient/URIException
>>>>> at
>>>>> org.elasticsearch.hadoop.hive.HiveUtils.structObjectInspector(HiveUtils.java:57)
>>>>> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:82)
>>>>> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:97)
>>>>>
>>>>> I'm quite puzzled because commons-httpclient is supposed to be on the
>>>>> hive-client classpath :
>>>>>
>>>>> # ls -l /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
>>>>> -rw-r--r-- 1 root root 279781 Mar 31 20:26
>>>>> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
>>>>> # ls -l /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
>>>>> -rw-r--r-- 1 root root 279781 Mar 31 20:26
>>>>> /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
>>>>>
>>>>> What am I missing ?
>>>>>
>>>>> Thanks a lot for your help,
>>>>>
>>>>> Kind regards,
>>>>>
>>>>> Erwan
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>
>
> --
> Nitin Pawar
>
Re: Hive and elasticsearch-hadoop-2.1.0 : NoClassDefFoundError
Posted by Nitin Pawar <ni...@gmail.com>.
I am using 2.2.4-2.2 but did not get any error.
can you check what all services are installed on the node where hive client
is running
On Mon, Jun 29, 2015 at 7:18 PM, Erwan Queffélec <er...@gmail.com>
wrote:
> Hi Nitin,
>
> No, I didn't do such a thing. I'm using the stock 0.14 version from HDP
> 2.2.4 (said 2.2.6 earlier but that was wrong)
>
> # hive --version
> Hive 0.14.0.2.2.4.2-2
> Subversion
> git://ip-10-0-0-5.ec2.internal/grid/0/jenkins/workspace/HDP-2.2.4.1-centos6/bigtop/build/hive/rpm/BUILD/hive-0.14.0.2.2.4.2
> -r 115d99896f5a4a81e7d91e052e8d38d7436b78d4
> Compiled by jenkins on Tue Mar 31 16:26:33 EDT 2015
> From source with checksum 1f34a1d4e566c3e801582862ed85ee93
>
> Thanks for taking the time.
>
> Kind regards,
>
> Erwan
>
> On Mon, Jun 29, 2015 at 3:44 PM, Nitin Pawar <ni...@gmail.com>
> wrote:
>
>> by any chance you built hive yourself ?
>>
>> On Mon, Jun 29, 2015 at 7:11 PM, Erwan Queffélec <
>> erwan.queffelec@gmail.com> wrote:
>>
>>> Additional info : it works when I manually add the jar with ADD JAR
>>> <file>;
>>>
>>> hive> ADD JAR '
>>> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar'
>>>
>>> I'm quite new to hive and hadoop in general. This is my first post on
>>> this mailing list, so please excuse me if the folowing question has been
>>> asked & answered over and over again :
>>>
>>> Perhaps I'm a bit naíve, but I though that hive Custom SerDe/UD*Fs were
>>> able to access everything already on the Hive classpath. Was it just a
>>> dream ?
>>>
>>> I would greatly appreciate some pointers, thanks to anyone who might be
>>> able to help !
>>>
>>> Best regards,
>>>
>>> Erwan
>>>
>>>
>>>
>>> On Mon, Jun 29, 2015 at 2:30 PM, Erwan Queffélec <
>>> erwan.queffelec@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm running HortonWorks HDP 2.2.6, hive 0.14 alongside an ElasticSearch
>>>> cluster
>>>>
>>>> For some reason Hive can't seem to connect to my ES cluster Using the
>>>> ES SerDe (I'm using elasticsearch-hadoop-2.1.0.jar). When time comes for my
>>>> job to insert the my query result in an ES resources, I get this stacktrace:
>>>>
>>>> NoClassDefFoundError: org/apache/commons/httpclient/URIException
>>>> at
>>>> org.elasticsearch.hadoop.hive.HiveUtils.structObjectInspector(HiveUtils.java:57)
>>>> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:82)
>>>> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:97)
>>>>
>>>> I'm quite puzzled because commons-httpclient is supposed to be on the
>>>> hive-client classpath :
>>>>
>>>> # ls -l /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
>>>> -rw-r--r-- 1 root root 279781 Mar 31 20:26
>>>> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
>>>> # ls -l /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
>>>> -rw-r--r-- 1 root root 279781 Mar 31 20:26
>>>> /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
>>>>
>>>> What am I missing ?
>>>>
>>>> Thanks a lot for your help,
>>>>
>>>> Kind regards,
>>>>
>>>> Erwan
>>>>
>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
--
Nitin Pawar
Re: Hive and elasticsearch-hadoop-2.1.0 : NoClassDefFoundError
Posted by Erwan Queffélec <er...@gmail.com>.
Hi Nitin,
No, I didn't do such a thing. I'm using the stock 0.14 version from HDP
2.2.4 (said 2.2.6 earlier but that was wrong)
# hive --version
Hive 0.14.0.2.2.4.2-2
Subversion
git://ip-10-0-0-5.ec2.internal/grid/0/jenkins/workspace/HDP-2.2.4.1-centos6/bigtop/build/hive/rpm/BUILD/hive-0.14.0.2.2.4.2
-r 115d99896f5a4a81e7d91e052e8d38d7436b78d4
Compiled by jenkins on Tue Mar 31 16:26:33 EDT 2015
>From source with checksum 1f34a1d4e566c3e801582862ed85ee93
Thanks for taking the time.
Kind regards,
Erwan
On Mon, Jun 29, 2015 at 3:44 PM, Nitin Pawar <ni...@gmail.com>
wrote:
> by any chance you built hive yourself ?
>
> On Mon, Jun 29, 2015 at 7:11 PM, Erwan Queffélec <
> erwan.queffelec@gmail.com> wrote:
>
>> Additional info : it works when I manually add the jar with ADD JAR
>> <file>;
>>
>> hive> ADD JAR '
>> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar'
>>
>> I'm quite new to hive and hadoop in general. This is my first post on
>> this mailing list, so please excuse me if the folowing question has been
>> asked & answered over and over again :
>>
>> Perhaps I'm a bit naíve, but I though that hive Custom SerDe/UD*Fs were
>> able to access everything already on the Hive classpath. Was it just a
>> dream ?
>>
>> I would greatly appreciate some pointers, thanks to anyone who might be
>> able to help !
>>
>> Best regards,
>>
>> Erwan
>>
>>
>>
>> On Mon, Jun 29, 2015 at 2:30 PM, Erwan Queffélec <
>> erwan.queffelec@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I'm running HortonWorks HDP 2.2.6, hive 0.14 alongside an ElasticSearch
>>> cluster
>>>
>>> For some reason Hive can't seem to connect to my ES cluster Using the ES
>>> SerDe (I'm using elasticsearch-hadoop-2.1.0.jar). When time comes for my
>>> job to insert the my query result in an ES resources, I get this stacktrace:
>>>
>>> NoClassDefFoundError: org/apache/commons/httpclient/URIException
>>> at
>>> org.elasticsearch.hadoop.hive.HiveUtils.structObjectInspector(HiveUtils.java:57)
>>> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:82)
>>> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:97)
>>>
>>> I'm quite puzzled because commons-httpclient is supposed to be on the
>>> hive-client classpath :
>>>
>>> # ls -l /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
>>> -rw-r--r-- 1 root root 279781 Mar 31 20:26
>>> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
>>> # ls -l /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
>>> -rw-r--r-- 1 root root 279781 Mar 31 20:26
>>> /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
>>>
>>> What am I missing ?
>>>
>>> Thanks a lot for your help,
>>>
>>> Kind regards,
>>>
>>> Erwan
>>>
>>
>>
>
>
> --
> Nitin Pawar
>
Re: Hive and elasticsearch-hadoop-2.1.0 : NoClassDefFoundError
Posted by Nitin Pawar <ni...@gmail.com>.
by any chance you built hive yourself ?
On Mon, Jun 29, 2015 at 7:11 PM, Erwan Queffélec <er...@gmail.com>
wrote:
> Additional info : it works when I manually add the jar with ADD JAR <file>;
>
> hive> ADD JAR '
> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar'
>
> I'm quite new to hive and hadoop in general. This is my first post on this
> mailing list, so please excuse me if the folowing question has been asked &
> answered over and over again :
>
> Perhaps I'm a bit naíve, but I though that hive Custom SerDe/UD*Fs were
> able to access everything already on the Hive classpath. Was it just a
> dream ?
>
> I would greatly appreciate some pointers, thanks to anyone who might be
> able to help !
>
> Best regards,
>
> Erwan
>
>
>
> On Mon, Jun 29, 2015 at 2:30 PM, Erwan Queffélec <
> erwan.queffelec@gmail.com> wrote:
>
>> Hello,
>>
>> I'm running HortonWorks HDP 2.2.6, hive 0.14 alongside an ElasticSearch
>> cluster
>>
>> For some reason Hive can't seem to connect to my ES cluster Using the ES
>> SerDe (I'm using elasticsearch-hadoop-2.1.0.jar). When time comes for my
>> job to insert the my query result in an ES resources, I get this stacktrace:
>>
>> NoClassDefFoundError: org/apache/commons/httpclient/URIException
>> at
>> org.elasticsearch.hadoop.hive.HiveUtils.structObjectInspector(HiveUtils.java:57)
>> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:82)
>> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:97)
>>
>> I'm quite puzzled because commons-httpclient is supposed to be on the
>> hive-client classpath :
>>
>> # ls -l /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
>> -rw-r--r-- 1 root root 279781 Mar 31 20:26
>> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
>> # ls -l /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
>> -rw-r--r-- 1 root root 279781 Mar 31 20:26
>> /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
>>
>> What am I missing ?
>>
>> Thanks a lot for your help,
>>
>> Kind regards,
>>
>> Erwan
>>
>
>
--
Nitin Pawar
Re: Hive and elasticsearch-hadoop-2.1.0 : NoClassDefFoundError
Posted by Erwan Queffélec <er...@gmail.com>.
Additional info : it works when I manually add the jar with ADD JAR <file>;
hive> ADD JAR '
/usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar'
I'm quite new to hive and hadoop in general. This is my first post on this
mailing list, so please excuse me if the folowing question has been asked &
answered over and over again :
Perhaps I'm a bit naíve, but I though that hive Custom SerDe/UD*Fs were
able to access everything already on the Hive classpath. Was it just a
dream ?
I would greatly appreciate some pointers, thanks to anyone who might be
able to help !
Best regards,
Erwan
On Mon, Jun 29, 2015 at 2:30 PM, Erwan Queffélec <er...@gmail.com>
wrote:
> Hello,
>
> I'm running HortonWorks HDP 2.2.6, hive 0.14 alongside an ElasticSearch
> cluster
>
> For some reason Hive can't seem to connect to my ES cluster Using the ES
> SerDe (I'm using elasticsearch-hadoop-2.1.0.jar). When time comes for my
> job to insert the my query result in an ES resources, I get this stacktrace:
>
> NoClassDefFoundError: org/apache/commons/httpclient/URIException
> at
> org.elasticsearch.hadoop.hive.HiveUtils.structObjectInspector(HiveUtils.java:57)
> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:82)
> at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:97)
>
> I'm quite puzzled because commons-httpclient is supposed to be on the
> hive-client classpath :
>
> # ls -l /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
> -rw-r--r-- 1 root root 279781 Mar 31 20:26
> /usr/hdp/current/hive-server2/lib/commons-httpclient-3.0.1.jar
> # ls -l /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
> -rw-r--r-- 1 root root 279781 Mar 31 20:26
> /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
>
> What am I missing ?
>
> Thanks a lot for your help,
>
> Kind regards,
>
> Erwan
>