You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jerrick Hoang <je...@gmail.com> on 2015/08/26 08:27:06 UTC
HiveMetaStoreClient
Hi all,
I want to interact with HiveMetaStore table from code and was looking at
http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
, was wondering if this is the correct way to do this or should I use a
jdbc client. If HiveMetaStoreClient is the right way then how do I create a
client given a hive-site.xml,
Thanks,
J
Re: HiveMetaStoreClient
Posted by Jörn Franke <jo...@gmail.com>.
What about using the hcatalog apis?
Le mer. 26 août 2015 à 8:27, Jerrick Hoang <je...@gmail.com> a
écrit :
> Hi all,
>
> I want to interact with HiveMetaStore table from code and was looking at
> http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
> , was wondering if this is the correct way to do this or should I use a
> jdbc client. If HiveMetaStoreClient is the right way then how do I create a
> client given a hive-site.xml,
>
> Thanks,
> J
>
Re: HiveMetaStoreClient
Posted by Dr Mich Talebzadeh <mi...@peridale.co.uk>.
Personally from my experience having your hive metastore on Oracle offers
the best concurrency. Oracle locking and concurrency is best suited for
metastore to avoid deadlocks and excessive serialisation.
On 26/8/2015, "Subroto Sanyal" <ss...@datameer.com> wrote:
>hi Jerrick
>
>Yes, thats the correct way to interact with HiveMetastore(provided you are
>only interested to play around metadata and Hive Metastore Service is
>accessible from the java client).
>Further, you can try the Hive JDBC driver or Hive Server2 thrift API
>(couldn't find the documentation for this but, it works for me) to access
>the metadata and physical data.
>
>On Wed, Aug 26, 2015 at 9:07 AM, Dr Mich Talebzadeh <mi...@peridale.co.uk>
>wrote:
>
>>
>> What is your hive metastore is on?
>>
>> If it is on an RDBMS, then you can simply use client/JDBC tools to access
>> the metastore. Pretty simple.
>>
>> HTH,
>>
>> On 26/8/2015, "Jerrick Hoang" <je...@gmail.com> wrote:
>>
>> >Hi all,
>> >
>> >I want to interact with HiveMetaStore table from code and was looking at
>> >
>> http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
>> >, was wondering if this is the correct way to do this or should I use a
>> >jdbc client. If HiveMetaStoreClient is the right way then how do I create
>> a
>> >client given a hive-site.xml,
>> >
>> >Thanks,
>> >J
>> >
>> >
>>
>
>
Re: HiveMetaStoreClient
Posted by Subroto Sanyal <ss...@datameer.com>.
hi Jerrick
Yes, thats the correct way to interact with HiveMetastore(provided you are
only interested to play around metadata and Hive Metastore Service is
accessible from the java client).
Further, you can try the Hive JDBC driver or Hive Server2 thrift API
(couldn't find the documentation for this but, it works for me) to access
the metadata and physical data.
On Wed, Aug 26, 2015 at 9:07 AM, Dr Mich Talebzadeh <mi...@peridale.co.uk>
wrote:
>
> What is your hive metastore is on?
>
> If it is on an RDBMS, then you can simply use client/JDBC tools to access
> the metastore. Pretty simple.
>
> HTH,
>
> On 26/8/2015, "Jerrick Hoang" <je...@gmail.com> wrote:
>
> >Hi all,
> >
> >I want to interact with HiveMetaStore table from code and was looking at
> >
> http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
> >, was wondering if this is the correct way to do this or should I use a
> >jdbc client. If HiveMetaStoreClient is the right way then how do I create
> a
> >client given a hive-site.xml,
> >
> >Thanks,
> >J
> >
> >
>
Re: HiveMetaStoreClient
Posted by Jörn Franke <jo...@gmail.com>.
Why not use hcatalog web service api?
Le mer. 26 août 2015 à 18:44, Jerrick Hoang <je...@gmail.com> a
écrit :
> Ok, I'm super confused now. The hive metastore is a RDBMS database. I
> totally agree that I shouldn't access it directly via jdbc. So what about
> using this class
> http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
> ? Is it something that can only be used
>
> If not I can definitely use the thrift api. Just wanted to ask this before
> going the thrift route because this class seems like the perfect tool for
> me,
>
> Thanks
>
> On Wed, Aug 26, 2015 at 8:33 AM, Alan Gates <al...@gmail.com> wrote:
>
>> Accessing the RDBMS directly isn't recommended. We don't view the table
>> layout as part of the public API and sometimes change things between
>> releases. With the thrift API we at least try to keep it backwards
>> compatible.
>>
>> Alan.
>>
>> Dr Mich Talebzadeh <mi...@peridale.co.uk>
>> August 26, 2015 at 0:07
>> What is your hive metastore is on?
>>
>> If it is on an RDBMS, then you can simply use client/JDBC tools to access
>> the metastore. Pretty simple.
>>
>> HTH,
>>
>> Jerrick Hoang <je...@gmail.com>
>> August 25, 2015 at 23:27
>> Hi all,
>>
>> I want to interact with HiveMetaStore table from code and was looking at
>> http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
>> , was wondering if this is the correct way to do this or should I use a
>> jdbc client. If HiveMetaStoreClient is the right way then how do I create a
>> client given a hive-site.xml,
>>
>> Thanks,
>> J
>>
>>
>
Re: HiveMetaStoreClient
Posted by Nathan Bamford <na...@redpoint.net>.
Hi Jerrick,
I'm sure someone more knowledgeable will correct me if I'm wrong, but I believe that class operates over the thrift api.
Nathan
________________________________
From: Jerrick Hoang <je...@gmail.com>
Sent: Wednesday, August 26, 2015 9:44 AM
To: user@hive.apache.org
Subject: Re: HiveMetaStoreClient
Ok, I'm super confused now. The hive metastore is a RDBMS database. I totally agree that I shouldn't access it directly via jdbc. So what about using this class http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html ? Is it something that can only be used
If not I can definitely use the thrift api. Just wanted to ask this before going the thrift route because this class seems like the perfect tool for me,
Thanks
On Wed, Aug 26, 2015 at 8:33 AM, Alan Gates <al...@gmail.com>> wrote:
Accessing the RDBMS directly isn't recommended. We don't view the table layout as part of the public API and sometimes change things between releases. With the thrift API we at least try to keep it backwards compatible.
Alan.
[cid:part1.07050003.08090809@gmail.com]
Dr Mich Talebzadeh<ma...@peridale.co.uk>
August 26, 2015 at 0:07
What is your hive metastore is on?
If it is on an RDBMS, then you can simply use client/JDBC tools to access
the metastore. Pretty simple.
HTH,
[cid:part1.07050003.08090809@gmail.com]
Jerrick Hoang<ma...@gmail.com>
August 25, 2015 at 23:27
Hi all,
I want to interact with HiveMetaStore table from code and was looking at http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html , was wondering if this is the correct way to do this or should I use a jdbc client. If HiveMetaStoreClient is the right way then how do I create a client given a hive-site.xml,
Thanks,
J
Re: HiveMetaStoreClient
Posted by Jerrick Hoang <je...@gmail.com>.
Hi Subroto, thanks for the suggestion, so I've read the api and looks like
setVar is the better way to go for me so I did this
HiveConf conf = new HiveConf();
conf.setVar(HiveConf.ConfVars.METASTOREURIS, "some-uri");
return new HiveMetaStoreClient(conf);
but I keep getting ` error: cannot access Configuration`. I have no idea
why this is happening, just wanted to point my HiveMetaStoreClient to my
HiveMetaStore service,
Would really appreciate if someone can help me out, thanks!
On Wed, Aug 26, 2015 at 11:31 AM, Subroto Sanyal <ss...@datameer.com>
wrote:
> Try something like:
> HiveConf hiveConf;
> hiveConf=new HiveConf();
> hiveConf.addResource("/etc/hive/conf/hive-default.xml");
> HiveMetaStoreClient hiveClient;
> try {
> hiveClient = new HiveMetaStoreClient(hiveConf);
> // use the client;
> } catch (Exception exp) {
> exp.printStackTrace();
> }
>
> On Wed, Aug 26, 2015 at 8:07 PM, Jerrick Hoang <je...@gmail.com>
> wrote:
>
>> Thanks all, that was super quick and helpful! Can someone give me a quick
>> code example of how to create a HiveMetaStoreClient object from
>> configurations in hive-site.xml, or how would I create a
>> HiveMetaStoreClient object given the uri and the credentials to talk to the
>> thrift server?
>>
>> On Wed, Aug 26, 2015 at 10:20 AM, Alan Gates <al...@gmail.com>
>> wrote:
>>
>>> Yes, sorry, I wasn't clear. As others have stated, HiveMetaStoreClient
>>> uses the thrift API. So using that should be fine.
>>>
>>> Alan.
>>>
>>> Subroto Sanyal <ss...@datameer.com>
>>> August 26, 2015 at 10:07
>>> Hi Jerrick,
>>> HiveMetastoreClient will be using thrift communication internally via
>>> the connection. IMO It is no different from using thrift api.
>>>
>>> Cheers,
>>> Subroto Sanyal
>>>
>>>
>>> Jerrick Hoang <je...@gmail.com>
>>> August 26, 2015 at 9:44
>>> Ok, I'm super confused now. The hive metastore is a RDBMS database. I
>>> totally agree that I shouldn't access it directly via jdbc. So what about
>>> using this class
>>> http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
>>> ? Is it something that can only be used
>>>
>>> If not I can definitely use the thrift api. Just wanted to ask this
>>> before going the thrift route because this class seems like the perfect
>>> tool for me,
>>>
>>> Thanks
>>>
>>>
>>>
>>
>
Re: HiveMetaStoreClient
Posted by Subroto Sanyal <ss...@datameer.com>.
Try something like:
HiveConf hiveConf;
hiveConf=new HiveConf();
hiveConf.addResource("/etc/hive/conf/hive-default.xml");
HiveMetaStoreClient hiveClient;
try {
hiveClient = new HiveMetaStoreClient(hiveConf);
// use the client;
} catch (Exception exp) {
exp.printStackTrace();
}
On Wed, Aug 26, 2015 at 8:07 PM, Jerrick Hoang <je...@gmail.com>
wrote:
> Thanks all, that was super quick and helpful! Can someone give me a quick
> code example of how to create a HiveMetaStoreClient object from
> configurations in hive-site.xml, or how would I create a
> HiveMetaStoreClient object given the uri and the credentials to talk to the
> thrift server?
>
> On Wed, Aug 26, 2015 at 10:20 AM, Alan Gates <al...@gmail.com> wrote:
>
>> Yes, sorry, I wasn't clear. As others have stated, HiveMetaStoreClient
>> uses the thrift API. So using that should be fine.
>>
>> Alan.
>>
>> Subroto Sanyal <ss...@datameer.com>
>> August 26, 2015 at 10:07
>> Hi Jerrick,
>> HiveMetastoreClient will be using thrift communication internally via the
>> connection. IMO It is no different from using thrift api.
>>
>> Cheers,
>> Subroto Sanyal
>>
>>
>> Jerrick Hoang <je...@gmail.com>
>> August 26, 2015 at 9:44
>> Ok, I'm super confused now. The hive metastore is a RDBMS database. I
>> totally agree that I shouldn't access it directly via jdbc. So what about
>> using this class
>> http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
>> ? Is it something that can only be used
>>
>> If not I can definitely use the thrift api. Just wanted to ask this
>> before going the thrift route because this class seems like the perfect
>> tool for me,
>>
>> Thanks
>>
>>
>>
>
Re: HiveMetaStoreClient
Posted by Jerrick Hoang <je...@gmail.com>.
Thanks all, that was super quick and helpful! Can someone give me a quick
code example of how to create a HiveMetaStoreClient object from
configurations in hive-site.xml, or how would I create a
HiveMetaStoreClient object given the uri and the credentials to talk to the
thrift server?
On Wed, Aug 26, 2015 at 10:20 AM, Alan Gates <al...@gmail.com> wrote:
> Yes, sorry, I wasn't clear. As others have stated, HiveMetaStoreClient
> uses the thrift API. So using that should be fine.
>
> Alan.
>
> Subroto Sanyal <ss...@datameer.com>
> August 26, 2015 at 10:07
> Hi Jerrick,
> HiveMetastoreClient will be using thrift communication internally via the
> connection. IMO It is no different from using thrift api.
>
> Cheers,
> Subroto Sanyal
>
>
> Jerrick Hoang <je...@gmail.com>
> August 26, 2015 at 9:44
> Ok, I'm super confused now. The hive metastore is a RDBMS database. I
> totally agree that I shouldn't access it directly via jdbc. So what about
> using this class
> http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
> ? Is it something that can only be used
>
> If not I can definitely use the thrift api. Just wanted to ask this before
> going the thrift route because this class seems like the perfect tool for
> me,
>
> Thanks
>
>
>
Re: HiveMetaStoreClient
Posted by Alan Gates <al...@gmail.com>.
Yes, sorry, I wasn't clear. As others have stated, HiveMetaStoreClient
uses the thrift API. So using that should be fine.
Alan.
> Subroto Sanyal <ma...@datameer.com>
> August 26, 2015 at 10:07
> Hi Jerrick,
> HiveMetastoreClient will be using thrift communication internally via
> the connection. IMO It is no different from using thrift api.
>
> Cheers,
> Subroto Sanyal
>
>
> Jerrick Hoang <ma...@gmail.com>
> August 26, 2015 at 9:44
> Ok, I'm super confused now. The hive metastore is a RDBMS database. I
> totally agree that I shouldn't access it directly via jdbc. So what
> about using this class
> http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
> ? Is it something that can only be used
>
> If not I can definitely use the thrift api. Just wanted to ask this
> before going the thrift route because this class seems like the
> perfect tool for me,
>
> Thanks
>
>
Re: HiveMetaStoreClient
Posted by Subroto Sanyal <ss...@datameer.com>.
Hi Jerrick,
HiveMetastoreClient will be using thrift communication internally via the
connection. IMO It is no different from using thrift api.
Cheers,
Subroto Sanyal
On Wed, Aug 26, 2015 at 6:44 PM, Jerrick Hoang <je...@gmail.com>
wrote:
> Ok, I'm super confused now. The hive metastore is a RDBMS database. I
> totally agree that I shouldn't access it directly via jdbc. So what about
> using this class
> http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
> ? Is it something that can only be used
>
> If not I can definitely use the thrift api. Just wanted to ask this before
> going the thrift route because this class seems like the perfect tool for
> me,
>
> Thanks
>
> On Wed, Aug 26, 2015 at 8:33 AM, Alan Gates <al...@gmail.com> wrote:
>
>> Accessing the RDBMS directly isn't recommended. We don't view the table
>> layout as part of the public API and sometimes change things between
>> releases. With the thrift API we at least try to keep it backwards
>> compatible.
>>
>> Alan.
>>
>> Dr Mich Talebzadeh <mi...@peridale.co.uk>
>> August 26, 2015 at 0:07
>> What is your hive metastore is on?
>>
>> If it is on an RDBMS, then you can simply use client/JDBC tools to access
>> the metastore. Pretty simple.
>>
>> HTH,
>>
>> Jerrick Hoang <je...@gmail.com>
>> August 25, 2015 at 23:27
>> Hi all,
>>
>> I want to interact with HiveMetaStore table from code and was looking at
>> http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
>> , was wondering if this is the correct way to do this or should I use a
>> jdbc client. If HiveMetaStoreClient is the right way then how do I create a
>> client given a hive-site.xml,
>>
>> Thanks,
>> J
>>
>>
>
Re: HiveMetaStoreClient
Posted by Jerrick Hoang <je...@gmail.com>.
Ok, I'm super confused now. The hive metastore is a RDBMS database. I
totally agree that I shouldn't access it directly via jdbc. So what about
using this class
http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
? Is it something that can only be used
If not I can definitely use the thrift api. Just wanted to ask this before
going the thrift route because this class seems like the perfect tool for
me,
Thanks
On Wed, Aug 26, 2015 at 8:33 AM, Alan Gates <al...@gmail.com> wrote:
> Accessing the RDBMS directly isn't recommended. We don't view the table
> layout as part of the public API and sometimes change things between
> releases. With the thrift API we at least try to keep it backwards
> compatible.
>
> Alan.
>
> Dr Mich Talebzadeh <mi...@peridale.co.uk>
> August 26, 2015 at 0:07
> What is your hive metastore is on?
>
> If it is on an RDBMS, then you can simply use client/JDBC tools to access
> the metastore. Pretty simple.
>
> HTH,
>
> Jerrick Hoang <je...@gmail.com>
> August 25, 2015 at 23:27
> Hi all,
>
> I want to interact with HiveMetaStore table from code and was looking at
> http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
> , was wondering if this is the correct way to do this or should I use a
> jdbc client. If HiveMetaStoreClient is the right way then how do I create a
> client given a hive-site.xml,
>
> Thanks,
> J
>
>
Re: HiveMetaStoreClient
Posted by Alan Gates <al...@gmail.com>.
Accessing the RDBMS directly isn't recommended. We don't view the table
layout as part of the public API and sometimes change things between
releases. With the thrift API we at least try to keep it backwards
compatible.
Alan.
> Dr Mich Talebzadeh <ma...@peridale.co.uk>
> August 26, 2015 at 0:07
> What is your hive metastore is on?
>
> If it is on an RDBMS, then you can simply use client/JDBC tools to access
> the metastore. Pretty simple.
>
> HTH,
>
> Jerrick Hoang <ma...@gmail.com>
> August 25, 2015 at 23:27
> Hi all,
>
> I want to interact with HiveMetaStore table from code and was looking
> at
> http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
> , was wondering if this is the correct way to do this or should I use
> a jdbc client. If HiveMetaStoreClient is the right way then how do I
> create a client given a hive-site.xml,
>
> Thanks,
> J
Re: HiveMetaStoreClient
Posted by Dr Mich Talebzadeh <mi...@peridale.co.uk>.
What is your hive metastore is on?
If it is on an RDBMS, then you can simply use client/JDBC tools to access
the metastore. Pretty simple.
HTH,
On 26/8/2015, "Jerrick Hoang" <je...@gmail.com> wrote:
>Hi all,
>
>I want to interact with HiveMetaStore table from code and was looking at
>http://hive.apache.org/javadocs/r0.13.1/api/metastore/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html
>, was wondering if this is the correct way to do this or should I use a
>jdbc client. If HiveMetaStoreClient is the right way then how do I create a
>client given a hive-site.xml,
>
>Thanks,
>J
>
>