Getting access to hadoop output from Hive JDBC session

[Alexander Kolbasov <ak...@conviva.com>]

Cross-posted from user@hive.apache.org

Hello,

I am switching from Hive 0.9 to Hive 0.12 and decided to start using Hive
metadata server mode. As it turns out, Hive1 JDBC driver connected as
"jdbc:hive://" only works via direct access to the metastore database. The
Hive2 driver connected as "jdbc:hive2://" does work with the remote Hive
metastore server, but there is another serious difference in behavior.
When I was using Hive1 driver I saw Hadoop output - the information about
Hive job ID and the usual Hadoop output showing percentages of map and
reduce done. The Hive2 driver silently waited for map/reduce to complete
and just produced the result.

As I can see, both Hive itself and beeline are able to get the same Hadoop
output as I was getting with Hive1 driver, so it should be somehow
possible but it isn't clear how they do this. Can someone suggest the way
to get Hadoop output with Hive2 JDBC driver?

Thanks for any help!

- Alex

Re: Getting access to hadoop output from Hive JDBC session

[Lefty Leverenz <le...@gmail.com>]

>
> If you were using a remote HS2, then you need a getLog api call, which is a
> work in progress in one of the jiras.
>

HIVE-4629 <https://issues.apache.org/jira/browse/HIVE-4629> "HS2 should
support an API to retrieve query logs"

-- Lefty


On Tue, Aug 12, 2014 at 10:45 PM, Thejas Nair <th...@hortonworks.com>
wrote:

> you are running HS2 in embedded mode, I think you should be able to get the
> hadoop output by setting the log4j settings appropriately.
> If you were using a remote HS2, then you need a getLog api call, which is a
> work in progress in one of the jiras.
>
>
>
> On Tue, Aug 12, 2014 at 6:17 PM, Alexander Kolbasov <ak...@conviva.com>
> wrote:
>
> > Cross-posted from user@hive.apache.org
> >
> > Hello,
> >
> > I am switching from Hive 0.9 to Hive 0.12 and decided to start using Hive
> > metadata server mode. As it turns out, Hive1 JDBC driver connected as
> > "jdbc:hive://" only works via direct access to the metastore database.
> The
> > Hive2 driver connected as "jdbc:hive2://" does work with the remote Hive
> > metastore server, but there is another serious difference in behavior.
> > When I was using Hive1 driver I saw Hadoop output - the information about
> > Hive job ID and the usual Hadoop output showing percentages of map and
> > reduce done. The Hive2 driver silently waited for map/reduce to complete
> > and just produced the result.
> >
> > As I can see, both Hive itself and beeline are able to get the same
> Hadoop
> > output as I was getting with Hive1 driver, so it should be somehow
> > possible but it isn't clear how they do this. Can someone suggest the way
> > to get Hadoop output with Hive2 JDBC driver?
> >
> > Thanks for any help!
> >
> > - Alex
> >
> >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Getting access to hadoop output from Hive JDBC session

[Lefty Leverenz <le...@gmail.com>]

>
> If you were using a remote HS2, then you need a getLog api call, which is a
> work in progress in one of the jiras.
>

HIVE-4629 <https://issues.apache.org/jira/browse/HIVE-4629> "HS2 should
support an API to retrieve query logs"

-- Lefty


On Tue, Aug 12, 2014 at 10:45 PM, Thejas Nair <th...@hortonworks.com>
wrote:

> you are running HS2 in embedded mode, I think you should be able to get the
> hadoop output by setting the log4j settings appropriately.
> If you were using a remote HS2, then you need a getLog api call, which is a
> work in progress in one of the jiras.
>
>
>
> On Tue, Aug 12, 2014 at 6:17 PM, Alexander Kolbasov <ak...@conviva.com>
> wrote:
>
> > Cross-posted from user@hive.apache.org
> >
> > Hello,
> >
> > I am switching from Hive 0.9 to Hive 0.12 and decided to start using Hive
> > metadata server mode. As it turns out, Hive1 JDBC driver connected as
> > "jdbc:hive://" only works via direct access to the metastore database.
> The
> > Hive2 driver connected as "jdbc:hive2://" does work with the remote Hive
> > metastore server, but there is another serious difference in behavior.
> > When I was using Hive1 driver I saw Hadoop output - the information about
> > Hive job ID and the usual Hadoop output showing percentages of map and
> > reduce done. The Hive2 driver silently waited for map/reduce to complete
> > and just produced the result.
> >
> > As I can see, both Hive itself and beeline are able to get the same
> Hadoop
> > output as I was getting with Hive1 driver, so it should be somehow
> > possible but it isn't clear how they do this. Can someone suggest the way
> > to get Hadoop output with Hive2 JDBC driver?
> >
> > Thanks for any help!
> >
> > - Alex
> >
> >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Getting access to hadoop output from Hive JDBC session

[Alexander Kolbasov <ak...@conviva.com>]

On 8/12/14, 7:45 PM, "Thejas Nair" <th...@hortonworks.com> wrote:

>you are running HS2 in embedded mode, I think you should be able to get
>the
>hadoop output by setting the log4j settings appropriately.
>If you were using a remote HS2, then you need a getLog api call, which is
>a
>work in progress in one of the jiras.

I do run it in embedded mode. I tried playing with log4j settings and it
didn¹t affect anything.

- Alex

>
>
>
>On Tue, Aug 12, 2014 at 6:17 PM, Alexander Kolbasov <ak...@conviva.com>
>wrote:
>
>> Cross-posted from user@hive.apache.org
>>
>> Hello,
>>
>> I am switching from Hive 0.9 to Hive 0.12 and decided to start using
>>Hive
>> metadata server mode. As it turns out, Hive1 JDBC driver connected as
>> "jdbc:hive://" only works via direct access to the metastore database.
>>The
>> Hive2 driver connected as "jdbc:hive2://" does work with the remote Hive
>> metastore server, but there is another serious difference in behavior.
>> When I was using Hive1 driver I saw Hadoop output - the information
>>about
>> Hive job ID and the usual Hadoop output showing percentages of map and
>> reduce done. The Hive2 driver silently waited for map/reduce to complete
>> and just produced the result.
>>
>> As I can see, both Hive itself and beeline are able to get the same
>>Hadoop
>> output as I was getting with Hive1 driver, so it should be somehow
>> possible but it isn't clear how they do this. Can someone suggest the
>>way
>> to get Hadoop output with Hive2 JDBC driver?
>>
>> Thanks for any help!
>>
>> - Alex
>>
>>
>>
>
>-- 
>CONFIDENTIALITY NOTICE
>NOTICE: This message is intended for the use of the individual or entity
>to 
>which it is addressed and may contain information that is confidential,
>privileged and exempt from disclosure under applicable law. If the reader
>of this message is not the intended recipient, you are hereby notified
>that 
>any printing, copying, dissemination, distribution, disclosure or
>forwarding of this communication is strictly prohibited. If you have
>received this communication in error, please contact the sender
>immediately 
>and delete it from your system. Thank You.
>

Re: Getting access to hadoop output from Hive JDBC session

[Thejas Nair <th...@hortonworks.com>]

you are running HS2 in embedded mode, I think you should be able to get the
hadoop output by setting the log4j settings appropriately.
If you were using a remote HS2, then you need a getLog api call, which is a
work in progress in one of the jiras.



On Tue, Aug 12, 2014 at 6:17 PM, Alexander Kolbasov <ak...@conviva.com>
wrote:

> Cross-posted from user@hive.apache.org
>
> Hello,
>
> I am switching from Hive 0.9 to Hive 0.12 and decided to start using Hive
> metadata server mode. As it turns out, Hive1 JDBC driver connected as
> "jdbc:hive://" only works via direct access to the metastore database. The
> Hive2 driver connected as "jdbc:hive2://" does work with the remote Hive
> metastore server, but there is another serious difference in behavior.
> When I was using Hive1 driver I saw Hadoop output - the information about
> Hive job ID and the usual Hadoop output showing percentages of map and
> reduce done. The Hive2 driver silently waited for map/reduce to complete
> and just produced the result.
>
> As I can see, both Hive itself and beeline are able to get the same Hadoop
> output as I was getting with Hive1 driver, so it should be somehow
> possible but it isn't clear how they do this. Can someone suggest the way
> to get Hadoop output with Hive2 JDBC driver?
>
> Thanks for any help!
>
> - Alex
>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.