Java api for connecting to hiveserver2

[Hanish Bansal <ha...@gmail.com>]

Hi All,

I am planning to use sql based authorization that is recently introduced in
hive 0.13.0.

I was using hive metastore client java api early which has direct apis for
all operations like grant, revoke etc.
But for using new authorization I ll have to use hiveserver2 and pass all
requests through hiveserver2. I came up with JDBC client which can be used
to connect hiveserver2. The restriction there is we must have to write SQL
statements.

I want to know is there any java api to connect hiveserver2, that have
direct java methods to perform operations and we don't need to write SQL
statements ??

Thanks,
Hanish

Re: Java api for connecting to hiveserver2

[Lefty Leverenz <le...@gmail.com>]

That's an older version of the HCatalog javadocs.  You can find current
javadocs here:  http://hive.apache.org/javadocs/r0.13.1/api/index.html.

-- Lefty

On Sun, Oct 12, 2014 at 3:05 PM, Suhas Gogate <vg...@pivotal.io> wrote:

> Edward/Hanish, this is a reasonable use case to justify Java API. I
> initially thought need for Java API to avoid knowing SQL :) but that you
> can't for any DML operations :)  So I guess for DDL operations you should
> transition to HCatalog API.. See the design document where first para
> explicitly advises it..
>
>
> https://cwiki.apache.org/confluence/display/HCATALOG/Design+Document+-+Java+APIs+for+HCatalog+DDL+Commands
>
> I have not played as much with HCatalog APIs yet and quick look tells me
> that it is not at par in terms of all DDL/auth operations yet!
>
> https://hive.apache.org/javadocs/hcat-r0.5.0/api/index.html
> (org.apache.hcatalog.api.HCatClient).
>
> --Suhas
>
>
> On Sat, Oct 11, 2014 at 2:52 PM, Hanish Bansal <
> hanish.bansal.agarwal@gmail.com> wrote:
>
>> Thanks all for response !!
>>
>> Totally agreed with Edward.
>>
>> Also the reason why we don't want to use jdbc client is: Writing sql
>> statement for everything is little bit complex rather than calling direct
>> java apis.
>> With java metastore client api it was quite easy.
>>
>> Regards,
>> Hanish
>>  On 12/10/2014 2:25 am, "Edward Capriolo" <ed...@gmail.com> wrote:
>>
>>> Actually I do not love the jdbc api for all things.
>>>
>>> I used to have several scripts that looped over all database, all
>>> tables, and looked at specific meta store properties or locations. These
>>> scripts were very small and to the point sometimes 5 line loops. Now I have
>>> to write things like this:
>>>
>>> ResultSet r = Statement.executeQuery("describe exteneded "+table)
>>> while (r.next){
>>> if (r.get(1).getString().equals("location")...
>>> ...
>>>
>>> It is more clunky then using the metastore api directly:
>>>
>>> hiveClient.getTable("database", "table").getSD().getLocation();
>>>
>>>
>>>
>>> On Sat, Oct 11, 2014 at 4:29 PM, Vaibhav Gumashta <
>>> vgumashta@hortonworks.com> wrote:
>>>
>>>> Hanish,
>>>>
>>>> I agree with Suhas and would strongly encourage you to use the JDBC API
>>>> for HiveServer2. HiveServer2 has a thrift api for the client-server RPC,
>>>> but that is *not* intended for end user consumption and could end up
>>>> breaking your current code in future.
>>>>
>>>> Is there any specific feature you are looking at which Hive's JDBC
>>>> driver doesn't implement?
>>>>
>>>> Thanks,
>>>> --Vaibhav
>>>>
>>>> On Sat, Oct 11, 2014 at 12:43 PM, Suhas Gogate <vg...@pivotal.io>
>>>> wrote:
>>>>
>>>>> Sorry Anish, but being database programmer in the past, I always used
>>>>> embedded SQL interface... I was wondering should we really need direct Java
>>>>> interface w/ HiveServer2? May be I am wrong, but would like to know your
>>>>> view on what are the limitations of using embedded SQL vs direct Java API.
>>>>>
>>>>> --Suhas
>>>>>
>>>>> On Sat, Oct 11, 2014 at 12:38 PM, Suhas Gogate <vg...@pivotal.io>
>>>>> wrote:
>>>>>
>>>>>> Hanish, this is interesting question and I also faced similar
>>>>>> limitation lately. Although as Hive getting more closer to relational model
>>>>>> with richer SQL interface (DDL/Authorization, DML) and HiveServer2 as a way
>>>>>> to invoke embedded SQL in Java, the real question is should Hive Metastore
>>>>>> Client (java) API should at all be used by user, rather all the existing
>>>>>> Hive Client interfaces be talking to Hive Metastore internally?
>>>>>>
>>>>>> --Suhas
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 10, 2014 at 9:21 PM, Hanish Bansal <
>>>>>> hanish.bansal.agarwal@gmail.com> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I am planning to use sql based authorization that is recently
>>>>>>> introduced in hive 0.13.0.
>>>>>>>
>>>>>>> I was using hive metastore client java api early which has direct
>>>>>>> apis for all operations like grant, revoke etc.
>>>>>>> But for using new authorization I ll have to use hiveserver2 and
>>>>>>> pass all requests through hiveserver2. I came up with JDBC client which can
>>>>>>> be used to connect hiveserver2. The restriction there is we must have to
>>>>>>> write SQL statements.
>>>>>>>
>>>>>>> I want to know is there any java api to connect hiveserver2, that
>>>>>>> have direct java methods to perform operations and we don't need to write
>>>>>>> SQL statements ??
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hanish
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>

Re: Java api for connecting to hiveserver2

[Suhas Gogate <vg...@pivotal.io>]

Edward/Hanish, this is a reasonable use case to justify Java API. I
initially thought need for Java API to avoid knowing SQL :) but that you
can't for any DML operations :)  So I guess for DDL operations you should
transition to HCatalog API.. See the design document where first para
explicitly advises it..

https://cwiki.apache.org/confluence/display/HCATALOG/Design+Document+-+Java+APIs+for+HCatalog+DDL+Commands

I have not played as much with HCatalog APIs yet and quick look tells me
that it is not at par in terms of all DDL/auth operations yet!

https://hive.apache.org/javadocs/hcat-r0.5.0/api/index.html
(org.apache.hcatalog.api.HCatClient).

--Suhas


On Sat, Oct 11, 2014 at 2:52 PM, Hanish Bansal <
hanish.bansal.agarwal@gmail.com> wrote:

> Thanks all for response !!
>
> Totally agreed with Edward.
>
> Also the reason why we don't want to use jdbc client is: Writing sql
> statement for everything is little bit complex rather than calling direct
> java apis.
> With java metastore client api it was quite easy.
>
> Regards,
> Hanish
>  On 12/10/2014 2:25 am, "Edward Capriolo" <ed...@gmail.com> wrote:
>
>> Actually I do not love the jdbc api for all things.
>>
>> I used to have several scripts that looped over all database, all tables,
>> and looked at specific meta store properties or locations. These scripts
>> were very small and to the point sometimes 5 line loops. Now I have to
>> write things like this:
>>
>> ResultSet r = Statement.executeQuery("describe exteneded "+table)
>> while (r.next){
>> if (r.get(1).getString().equals("location")...
>> ...
>>
>> It is more clunky then using the metastore api directly:
>>
>> hiveClient.getTable("database", "table").getSD().getLocation();
>>
>>
>>
>> On Sat, Oct 11, 2014 at 4:29 PM, Vaibhav Gumashta <
>> vgumashta@hortonworks.com> wrote:
>>
>>> Hanish,
>>>
>>> I agree with Suhas and would strongly encourage you to use the JDBC API
>>> for HiveServer2. HiveServer2 has a thrift api for the client-server RPC,
>>> but that is *not* intended for end user consumption and could end up
>>> breaking your current code in future.
>>>
>>> Is there any specific feature you are looking at which Hive's JDBC
>>> driver doesn't implement?
>>>
>>> Thanks,
>>> --Vaibhav
>>>
>>> On Sat, Oct 11, 2014 at 12:43 PM, Suhas Gogate <vg...@pivotal.io>
>>> wrote:
>>>
>>>> Sorry Anish, but being database programmer in the past, I always used
>>>> embedded SQL interface... I was wondering should we really need direct Java
>>>> interface w/ HiveServer2? May be I am wrong, but would like to know your
>>>> view on what are the limitations of using embedded SQL vs direct Java API.
>>>>
>>>> --Suhas
>>>>
>>>> On Sat, Oct 11, 2014 at 12:38 PM, Suhas Gogate <vg...@pivotal.io>
>>>> wrote:
>>>>
>>>>> Hanish, this is interesting question and I also faced similar
>>>>> limitation lately. Although as Hive getting more closer to relational model
>>>>> with richer SQL interface (DDL/Authorization, DML) and HiveServer2 as a way
>>>>> to invoke embedded SQL in Java, the real question is should Hive Metastore
>>>>> Client (java) API should at all be used by user, rather all the existing
>>>>> Hive Client interfaces be talking to Hive Metastore internally?
>>>>>
>>>>> --Suhas
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Oct 10, 2014 at 9:21 PM, Hanish Bansal <
>>>>> hanish.bansal.agarwal@gmail.com> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I am planning to use sql based authorization that is recently
>>>>>> introduced in hive 0.13.0.
>>>>>>
>>>>>> I was using hive metastore client java api early which has direct
>>>>>> apis for all operations like grant, revoke etc.
>>>>>> But for using new authorization I ll have to use hiveserver2 and pass
>>>>>> all requests through hiveserver2. I came up with JDBC client which can be
>>>>>> used to connect hiveserver2. The restriction there is we must have to write
>>>>>> SQL statements.
>>>>>>
>>>>>> I want to know is there any java api to connect hiveserver2, that
>>>>>> have direct java methods to perform operations and we don't need to write
>>>>>> SQL statements ??
>>>>>>
>>>>>> Thanks,
>>>>>> Hanish
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>

Re: Java api for connecting to hiveserver2

[Hanish Bansal <ha...@gmail.com>]

Thanks all for response !!

Totally agreed with Edward.

Also the reason why we don't want to use jdbc client is: Writing sql
statement for everything is little bit complex rather than calling direct
java apis.
With java metastore client api it was quite easy.

Regards,
Hanish
 On 12/10/2014 2:25 am, "Edward Capriolo" <ed...@gmail.com> wrote:

> Actually I do not love the jdbc api for all things.
>
> I used to have several scripts that looped over all database, all tables,
> and looked at specific meta store properties or locations. These scripts
> were very small and to the point sometimes 5 line loops. Now I have to
> write things like this:
>
> ResultSet r = Statement.executeQuery("describe exteneded "+table)
> while (r.next){
> if (r.get(1).getString().equals("location")...
> ...
>
> It is more clunky then using the metastore api directly:
>
> hiveClient.getTable("database", "table").getSD().getLocation();
>
>
>
> On Sat, Oct 11, 2014 at 4:29 PM, Vaibhav Gumashta <
> vgumashta@hortonworks.com> wrote:
>
>> Hanish,
>>
>> I agree with Suhas and would strongly encourage you to use the JDBC API
>> for HiveServer2. HiveServer2 has a thrift api for the client-server RPC,
>> but that is *not* intended for end user consumption and could end up
>> breaking your current code in future.
>>
>> Is there any specific feature you are looking at which Hive's JDBC driver
>> doesn't implement?
>>
>> Thanks,
>> --Vaibhav
>>
>> On Sat, Oct 11, 2014 at 12:43 PM, Suhas Gogate <vg...@pivotal.io>
>> wrote:
>>
>>> Sorry Anish, but being database programmer in the past, I always used
>>> embedded SQL interface... I was wondering should we really need direct Java
>>> interface w/ HiveServer2? May be I am wrong, but would like to know your
>>> view on what are the limitations of using embedded SQL vs direct Java API.
>>>
>>> --Suhas
>>>
>>> On Sat, Oct 11, 2014 at 12:38 PM, Suhas Gogate <vg...@pivotal.io>
>>> wrote:
>>>
>>>> Hanish, this is interesting question and I also faced similar
>>>> limitation lately. Although as Hive getting more closer to relational model
>>>> with richer SQL interface (DDL/Authorization, DML) and HiveServer2 as a way
>>>> to invoke embedded SQL in Java, the real question is should Hive Metastore
>>>> Client (java) API should at all be used by user, rather all the existing
>>>> Hive Client interfaces be talking to Hive Metastore internally?
>>>>
>>>> --Suhas
>>>>
>>>>
>>>>
>>>> On Fri, Oct 10, 2014 at 9:21 PM, Hanish Bansal <
>>>> hanish.bansal.agarwal@gmail.com> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I am planning to use sql based authorization that is recently
>>>>> introduced in hive 0.13.0.
>>>>>
>>>>> I was using hive metastore client java api early which has direct apis
>>>>> for all operations like grant, revoke etc.
>>>>> But for using new authorization I ll have to use hiveserver2 and pass
>>>>> all requests through hiveserver2. I came up with JDBC client which can be
>>>>> used to connect hiveserver2. The restriction there is we must have to write
>>>>> SQL statements.
>>>>>
>>>>> I want to know is there any java api to connect hiveserver2, that have
>>>>> direct java methods to perform operations and we don't need to write SQL
>>>>> statements ??
>>>>>
>>>>> Thanks,
>>>>> Hanish
>>>>>
>>>>
>>>>
>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>

Re: Java api for connecting to hiveserver2

[Edward Capriolo <ed...@gmail.com>]

Actually I do not love the jdbc api for all things.

I used to have several scripts that looped over all database, all tables,
and looked at specific meta store properties or locations. These scripts
were very small and to the point sometimes 5 line loops. Now I have to
write things like this:

ResultSet r = Statement.executeQuery("describe exteneded "+table)
while (r.next){
if (r.get(1).getString().equals("location")...
...

It is more clunky then using the metastore api directly:

hiveClient.getTable("database", "table").getSD().getLocation();



On Sat, Oct 11, 2014 at 4:29 PM, Vaibhav Gumashta <vgumashta@hortonworks.com
> wrote:

> Hanish,
>
> I agree with Suhas and would strongly encourage you to use the JDBC API
> for HiveServer2. HiveServer2 has a thrift api for the client-server RPC,
> but that is *not* intended for end user consumption and could end up
> breaking your current code in future.
>
> Is there any specific feature you are looking at which Hive's JDBC driver
> doesn't implement?
>
> Thanks,
> --Vaibhav
>
> On Sat, Oct 11, 2014 at 12:43 PM, Suhas Gogate <vg...@pivotal.io> wrote:
>
>> Sorry Anish, but being database programmer in the past, I always used
>> embedded SQL interface... I was wondering should we really need direct Java
>> interface w/ HiveServer2? May be I am wrong, but would like to know your
>> view on what are the limitations of using embedded SQL vs direct Java API.
>>
>> --Suhas
>>
>> On Sat, Oct 11, 2014 at 12:38 PM, Suhas Gogate <vg...@pivotal.io>
>> wrote:
>>
>>> Hanish, this is interesting question and I also faced similar limitation
>>> lately. Although as Hive getting more closer to relational model with
>>> richer SQL interface (DDL/Authorization, DML) and HiveServer2 as a way to
>>> invoke embedded SQL in Java, the real question is should Hive Metastore
>>> Client (java) API should at all be used by user, rather all the existing
>>> Hive Client interfaces be talking to Hive Metastore internally?
>>>
>>> --Suhas
>>>
>>>
>>>
>>> On Fri, Oct 10, 2014 at 9:21 PM, Hanish Bansal <
>>> hanish.bansal.agarwal@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I am planning to use sql based authorization that is recently
>>>> introduced in hive 0.13.0.
>>>>
>>>> I was using hive metastore client java api early which has direct apis
>>>> for all operations like grant, revoke etc.
>>>> But for using new authorization I ll have to use hiveserver2 and pass
>>>> all requests through hiveserver2. I came up with JDBC client which can be
>>>> used to connect hiveserver2. The restriction there is we must have to write
>>>> SQL statements.
>>>>
>>>> I want to know is there any java api to connect hiveserver2, that have
>>>> direct java methods to perform operations and we don't need to write SQL
>>>> statements ??
>>>>
>>>> Thanks,
>>>> Hanish
>>>>
>>>
>>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Java api for connecting to hiveserver2

[Vaibhav Gumashta <vg...@hortonworks.com>]

Hanish,

I agree with Suhas and would strongly encourage you to use the JDBC API for
HiveServer2. HiveServer2 has a thrift api for the client-server RPC, but
that is *not* intended for end user consumption and could end up breaking
your current code in future.

Is there any specific feature you are looking at which Hive's JDBC driver
doesn't implement?

Thanks,
--Vaibhav

On Sat, Oct 11, 2014 at 12:43 PM, Suhas Gogate <vg...@pivotal.io> wrote:

> Sorry Anish, but being database programmer in the past, I always used
> embedded SQL interface... I was wondering should we really need direct Java
> interface w/ HiveServer2? May be I am wrong, but would like to know your
> view on what are the limitations of using embedded SQL vs direct Java API.
>
> --Suhas
>
> On Sat, Oct 11, 2014 at 12:38 PM, Suhas Gogate <vg...@pivotal.io> wrote:
>
>> Hanish, this is interesting question and I also faced similar limitation
>> lately. Although as Hive getting more closer to relational model with
>> richer SQL interface (DDL/Authorization, DML) and HiveServer2 as a way to
>> invoke embedded SQL in Java, the real question is should Hive Metastore
>> Client (java) API should at all be used by user, rather all the existing
>> Hive Client interfaces be talking to Hive Metastore internally?
>>
>> --Suhas
>>
>>
>>
>> On Fri, Oct 10, 2014 at 9:21 PM, Hanish Bansal <
>> hanish.bansal.agarwal@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> I am planning to use sql based authorization that is recently introduced
>>> in hive 0.13.0.
>>>
>>> I was using hive metastore client java api early which has direct apis
>>> for all operations like grant, revoke etc.
>>> But for using new authorization I ll have to use hiveserver2 and pass
>>> all requests through hiveserver2. I came up with JDBC client which can be
>>> used to connect hiveserver2. The restriction there is we must have to write
>>> SQL statements.
>>>
>>> I want to know is there any java api to connect hiveserver2, that have
>>> direct java methods to perform operations and we don't need to write SQL
>>> statements ??
>>>
>>> Thanks,
>>> Hanish
>>>
>>
>>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Java api for connecting to hiveserver2

[Suhas Gogate <vg...@pivotal.io>]

Sorry Anish, but being database programmer in the past, I always used
embedded SQL interface... I was wondering should we really need direct Java
interface w/ HiveServer2? May be I am wrong, but would like to know your
view on what are the limitations of using embedded SQL vs direct Java API.

--Suhas

On Sat, Oct 11, 2014 at 12:38 PM, Suhas Gogate <vg...@pivotal.io> wrote:

> Hanish, this is interesting question and I also faced similar limitation
> lately. Although as Hive getting more closer to relational model with
> richer SQL interface (DDL/Authorization, DML) and HiveServer2 as a way to
> invoke embedded SQL in Java, the real question is should Hive Metastore
> Client (java) API should at all be used by user, rather all the existing
> Hive Client interfaces be talking to Hive Metastore internally?
>
> --Suhas
>
>
>
> On Fri, Oct 10, 2014 at 9:21 PM, Hanish Bansal <
> hanish.bansal.agarwal@gmail.com> wrote:
>
>> Hi All,
>>
>> I am planning to use sql based authorization that is recently introduced
>> in hive 0.13.0.
>>
>> I was using hive metastore client java api early which has direct apis
>> for all operations like grant, revoke etc.
>> But for using new authorization I ll have to use hiveserver2 and pass all
>> requests through hiveserver2. I came up with JDBC client which can be used
>> to connect hiveserver2. The restriction there is we must have to write SQL
>> statements.
>>
>> I want to know is there any java api to connect hiveserver2, that have
>> direct java methods to perform operations and we don't need to write SQL
>> statements ??
>>
>> Thanks,
>> Hanish
>>
>
>

Re: Java api for connecting to hiveserver2

[Suhas Gogate <vg...@pivotal.io>]

Hanish, this is interesting question and I also faced similar limitation
lately. Although as Hive getting more closer to relational model with
richer SQL interface (DDL/Authorization, DML) and HiveServer2 as a way to
invoke embedded SQL in Java, the real question is should Hive Metastore
Client (java) API should at all be used by user, rather all the existing
Hive Client interfaces be talking to Hive Metastore internally?

--Suhas



On Fri, Oct 10, 2014 at 9:21 PM, Hanish Bansal <
hanish.bansal.agarwal@gmail.com> wrote:

> Hi All,
>
> I am planning to use sql based authorization that is recently introduced
> in hive 0.13.0.
>
> I was using hive metastore client java api early which has direct apis for
> all operations like grant, revoke etc.
> But for using new authorization I ll have to use hiveserver2 and pass all
> requests through hiveserver2. I came up with JDBC client which can be used
> to connect hiveserver2. The restriction there is we must have to write SQL
> statements.
>
> I want to know is there any java api to connect hiveserver2, that have
> direct java methods to perform operations and we don't need to write SQL
> statements ??
>
> Thanks,
> Hanish
>