You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by David Morel <dm...@gmail.com> on 2013/01/04 00:16:26 UTC

Thrift Hive client for CDH 4.1 HiveServer2?

Hi all (and happy New Year!)

Is it possible to build a perl Thrift client for HiveServer2 (from
Cloudera's 4.1.x) ?

I'm following the instructions found here:
http://stackoverflow.com/questions/5289164/perl-thrift-client-to-hive

Downloaded Hive from Cloudera's site, then i'm a bit lost: where do I find
these thrift files that I need to build the perl libs? I have the thrift
compiler working ok, but thats as far as I got.

Any help would be most welcome

Thanks!

D.Morel

Re: Thrift Hive client for CDH 4.1 HiveServer2?

Posted by Carl Steinbach <cw...@apache.org>.
Hi,

The Thrift IDL file for HiveServer2 is located here:

service/if/cli_service.thrift

Here's what you need to do in order to build the HiveServer2 Perl Thrift
client using the Hive tarball from CDH4.1.x:

# Set THRIFT_HOME to point to the installation directory of Thrift 0.9.0
% export THRIFT_HOME=<...>

# Verify that THRIFT_HOME is set properly
% $THRIFT_HOME/bin/thrift -version
Thrift Version 0.9.0

# Download and unpack the hive-0.9.0-cdh4.1.2 tarball
% wget http://archive.cloudera.com/cdh4/cdh/4/hive-0.9.0-cdh4.1.2.tar.gz
% tar -xvf hive-0.9.0-cdh4.1.2.tar.gz
% cd hive-0.9.0-cdh4.1.2/src

# Generate the Perl Thrift libraries
% ant thriftif -Dthrift.home=$THRIFT_HOME -Dthrift.args="-I $THRIFT_HOME
--gen perl"

The previous command will result in the generation of the HiveServer2 Perl
library here:

service/src/gen/thrift/gen-perl/TCLIService.pm

In order to use this library you can start with the code on stackoverflow
and modify
it to reference TCLIService instead of ThriftHiveClient. Also note that the
TCLIService
Thrift service exposes a different set of functions than the
ThriftHiveClient service.
I recommend looking at the comments in TCLIService.thrift in order to
understand
the proper workflow.

Thanks.

Carl



On Sat, Jan 5, 2013 at 3:00 AM, David Morel <dm...@gmail.com> wrote:

> So that would probably be generated
> using src/service/if/cli_service.thrift instead of the older
> hive_service.thrift which i suppose is for hiveserver1.
> Compiled it, still getting errors that seem transport-related
>
> 13/01/04 23:02:22 ERROR server.TThreadPoolServer: Error occurred during
> processing of message.
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
>  at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:218)
> ...
>
> This is a bit tedious...
>
> D.Morel
>
>
> On Sat, Jan 5, 2013 at 10:27 AM, Jov <zh...@gmail.com> wrote:
>
>> here：
>> https://issues.apache.org/jira/browse/HIVE-2935
>> https://cwiki.apache.org/Hive/hiveserver2-thrift-api.html
>> HiveServer2 now is CDH extension.
>>
>> I think you can use find cmd to search the CDH src dir to find the
>> .thrift files.
>>
>>
>> 2013/1/5 David Morel <dm...@gmail.com>
>>
>>> On 4 Jan 2013, at 16:04, Jov wrote:
>>>
>>> they are in the src/service/if and src/metastore/if
>>>
>>> Cool. But these would be files for HiveServer, not HiveServer2 which has
>>> a different API, right? After finally generating the libs, it turns out
>>> they work fine on the old-style hive server, but produce this in
>>> hiveserver2's log: 13/01/04 20:09:11 ERROR server.TThreadPoolServer: Error
>>> occurred during processing of message. java.lang.RuntimeException:
>>> org.apache.thrift.transport.TTransportException at
>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:218)
>>> at
>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:170)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662) Caused by:
>>> org.apache.thrift.transport.TTransportException at
>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at
>>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>>> at
>>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:124)
>>> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>>> at
>>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:40)
>>> at
>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:215)
>>> ... 4 more Where should I start looking (meaning I haven't a clue)? Thanks!
>>> David
>>>
>>> 在 2013-1-4 上午7:16，"David Morel" <dm...@gmail.com>写道：
>>>
>>> Hi all (and happy New Year!) Is it possible to build a perl Thrift
>>> client for HiveServer2 (from Cloudera's 4.1.x) ? I'm following the
>>> instructions found here:
>>> http://stackoverflow.com/questions/5289164/perl-thrift-client-to-hiveDownloaded Hive from Cloudera's site, then i'm a bit lost: where do I find
>>> these thrift files that I need to build the perl libs? I have the thrift
>>> compiler working ok, but thats as far as I got.
>>>
>>>
>>
>>
>> --
>> jov
>> blog: http:amutu.com/blog
>>
>
>

Re: Thrift Hive client for CDH 4.1 HiveServer2?

Posted by David Morel <dm...@gmail.com>.
So that would probably be generated using src/service/if/cli_service.thrift
instead of the older hive_service.thrift which i suppose is for hiveserver1.
Compiled it, still getting errors that seem transport-related

13/01/04 23:02:22 ERROR server.TThreadPoolServer: Error occurred during
processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:218)
...

This is a bit tedious...

D.Morel


On Sat, Jan 5, 2013 at 10:27 AM, Jov <zh...@gmail.com> wrote:

> here：
> https://issues.apache.org/jira/browse/HIVE-2935
> https://cwiki.apache.org/Hive/hiveserver2-thrift-api.html
> HiveServer2 now is CDH extension.
>
> I think you can use find cmd to search the CDH src dir to find the .thrift
> files.
>
>
> 2013/1/5 David Morel <dm...@gmail.com>
>
>> On 4 Jan 2013, at 16:04, Jov wrote:
>>
>> they are in the src/service/if and src/metastore/if
>>
>> Cool. But these would be files for HiveServer, not HiveServer2 which has
>> a different API, right? After finally generating the libs, it turns out
>> they work fine on the old-style hive server, but produce this in
>> hiveserver2's log: 13/01/04 20:09:11 ERROR server.TThreadPoolServer: Error
>> occurred during processing of message. java.lang.RuntimeException:
>> org.apache.thrift.transport.TTransportException at
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:218)
>> at
>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:170)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662) Caused by:
>> org.apache.thrift.transport.TTransportException at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at
>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>> at
>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:124)
>> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>> at
>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:40)
>> at
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:215)
>> ... 4 more Where should I start looking (meaning I haven't a clue)? Thanks!
>> David
>>
>> 在 2013-1-4 上午7:16，"David Morel" <dm...@gmail.com>写道：
>>
>> Hi all (and happy New Year!) Is it possible to build a perl Thrift client
>> for HiveServer2 (from Cloudera's 4.1.x) ? I'm following the instructions
>> found here:
>> http://stackoverflow.com/questions/5289164/perl-thrift-client-to-hiveDownloaded Hive from Cloudera's site, then i'm a bit lost: where do I find
>> these thrift files that I need to build the perl libs? I have the thrift
>> compiler working ok, but thats as far as I got.
>>
>>
>
>
> --
> jov
> blog: http:amutu.com/blog
>

Re: Thrift Hive client for CDH 4.1 HiveServer2?

Posted by Jov <zh...@gmail.com>.
here：
https://issues.apache.org/jira/browse/HIVE-2935
https://cwiki.apache.org/Hive/hiveserver2-thrift-api.html
HiveServer2 now is CDH extension.

I think you can use find cmd to search the CDH src dir to find the .thrift
files.

2013/1/5 David Morel <dm...@gmail.com>

> On 4 Jan 2013, at 16:04, Jov wrote:
>
> they are in the src/service/if and src/metastore/if
>
> Cool. But these would be files for HiveServer, not HiveServer2 which has a
> different API, right? After finally generating the libs, it turns out they
> work fine on the old-style hive server, but produce this in hiveserver2's
> log: 13/01/04 20:09:11 ERROR server.TThreadPoolServer: Error occurred
> during processing of message. java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:218)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:170)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662) Caused by:
> org.apache.thrift.transport.TTransportException at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
> at
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:124)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:40)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:215)
> ... 4 more Where should I start looking (meaning I haven't a clue)? Thanks!
> David
>
> 在 2013-1-4 上午7:16，"David Morel" <dm...@gmail.com>写道：
>
> Hi all (and happy New Year!) Is it possible to build a perl Thrift client
> for HiveServer2 (from Cloudera's 4.1.x) ? I'm following the instructions
> found here:
> http://stackoverflow.com/questions/5289164/perl-thrift-client-to-hiveDownloaded Hive from Cloudera's site, then i'm a bit lost: where do I find
> these thrift files that I need to build the perl libs? I have the thrift
> compiler working ok, but thats as far as I got.
>
>


-- 
jov
blog: http:amutu.com/blog

Re: Thrift Hive client for CDH 4.1 HiveServer2?

Posted by David Morel <dm...@gmail.com>.
On 4 Jan 2013, at 16:04, Jov wrote:

they are in the src/service/if and src/metastore/if

Cool. But these would be files for HiveServer, not HiveServer2 which has a
different API, right? After finally generating the libs, it turns out they
work fine on the old-style hive server, but produce this in hiveserver2's
log: 13/01/04 20:09:11 ERROR server.TThreadPoolServer: Error occurred
during processing of message. java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:218)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:170)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Caused by:
org.apache.thrift.transport.TTransportException at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
at
org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:124)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:40)
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:215)
... 4 more Where should I start looking (meaning I haven't a clue)? Thanks!
David

在 2013-1-4 上午7:16，"David Morel" <dm...@gmail.com>写道：

Hi all (and happy New Year!) Is it possible to build a perl Thrift client
for HiveServer2 (from Cloudera's 4.1.x) ? I'm following the instructions
found here:
http://stackoverflow.com/questions/5289164/perl-thrift-client-to-hiveDownloaded
Hive from Cloudera's site, then i'm a bit lost: where do I find
these thrift files that I need to build the perl libs? I have the thrift
compiler working ok, but thats as far as I got.

Re: Thrift Hive client for CDH 4.1 HiveServer2?

Posted by Jov <zh...@gmail.com>.
they are in the src/service/if and src/metastore/if
在 2013-1-4 上午7:16，"David Morel" <dm...@gmail.com>写道：

> Hi all (and happy New Year!)
>
> Is it possible to build a perl Thrift client for HiveServer2 (from
> Cloudera's 4.1.x) ?
>
> I'm following the instructions found here:
> http://stackoverflow.com/questions/5289164/perl-thrift-client-to-hive
>
> Downloaded Hive from Cloudera's site, then i'm a bit lost: where do I find
> these thrift files that I need to build the perl libs? I have the thrift
> compiler working ok, but thats as far as I got.
>
> Any help would be most welcome
>
> Thanks!
>
> D.Morel
>