You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Schubert Zhang <zs...@gmail.com> on 2009/09/04 11:17:28 UTC
Does current Hive trunk support hadoop-0.20.0?
Does current Hive trunk support hadoop-0.20.0?
Re: Does current Hive trunk support hadoop-0.20.0?
Posted by Schubert Zhang <zs...@gmail.com>.
@Zheng,
We can build hive for hadoop-0.20.0.
But I checked the trunk code, and find the current code does not be updated
to use new hadoop-0.20.0 APIs, expecially for new MapReduce package.
Schubert
On Sat, Sep 5, 2009 at 3:54 AM, Vijay <te...@gmail.com> wrote:
> Just a follow up to that: is that trunk only or does the 0.4 branch support
> hadoop-0.20.0 as well?
>
>
> On Fri, Sep 4, 2009 at 10:13 AM, Zheng Shao <zs...@gmail.com> wrote:
>
>> Yes.
>>
>> Zheng
>>
>> On Fri, Sep 4, 2009 at 2:17 AM, Schubert Zhang <zs...@gmail.com> wrote:
>>
>>>
>>> Does current Hive trunk support hadoop-0.20.0?
>>>
>>
>>
>>
>> --
>> Yours,
>> Zheng
>>
>
>
Re: MetaStoreClient not getting cached
Posted by Prasad Chakka <pc...@facebook.com>.
Thanks David. This will be in trunk & 0.4 soon.
Prasad
________________________________
From: David Lerman <dl...@videoegg.com>
Reply-To: <hi...@hadoop.apache.org>
Date: Fri, 4 Sep 2009 13:41:43 -0700
To: <hi...@hadoop.apache.org>
Subject: Re: MetaStoreClient not getting cached
Thanks Prasad, patch is available in HIVE-816.
On 9/4/09 4:24 PM, "Prasad Chakka" <pc...@facebook.com> wrote:
> Yeah, you are correct. It is a bug. It will be great if you can submit a
> patch.
>
> Prasad
>
>
>
> From: David Lerman <dl...@videoegg.com>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Fri, 4 Sep 2009 13:19:28 -0700
> To: <hi...@hadoop.apache.org>
> Subject: MetaStoreClient not getting cached
>
> Planning a query is taking a very long time, and during the pause, I see the
> following log lines hundreds of times in a row
>
> 2009-09-04 16:14:17,090 INFO metastore.HiveMetaStore
> (HiveMetaStore.java:getMS(125)) - 0: Opening raw store with implemenation
> class:org.apache.hadoop.hive.metastore.ObjectStore
> 2009-09-04 16:14:17,090
> INFO metastore.ObjectStore (ObjectStore.java:initialize(120)) -
> ObjectStore, initialize called
> 2009-09-04 16:14:17,090 INFO
> metastore.ObjectStore (ObjectStore.java:setConf(106)) - Initialized
> ObjectStore
>
> Going through the code, I see that the PartitionPruner is asking for
> metadata on every partition, and every call ends up calling
> org.apache.hadoop.hive.ql.metadata.Hive.getMSC():
>
> private IMetaStoreClient getMSC() throws MetaException {
> IMetaStoreClient msc = threadLocalMSC.get();
> if(msc == null) {
> msc = this.createMetaStoreClient();
> }
> return msc;
> }
>
> Each call ends up creating a new instance, because threadLocalMSC.set() is
> never called. Is there a reason this isn't:
>
> private IMetaStoreClient getMSC() throws MetaException {
> IMetaStoreClient msc = threadLocalMSC.get();
> if(msc == null) {
> msc = this.createMetaStoreClient();
> threadLocalMSC.set(msc); // CHANGE HERE!
> }
> return msc;
> }
>
> Thanks!
Re: MetaStoreClient not getting cached
Posted by David Lerman <dl...@videoegg.com>.
Thanks Prasad, patch is available in HIVE-816.
On 9/4/09 4:24 PM, "Prasad Chakka" <pc...@facebook.com> wrote:
> Yeah, you are correct. It is a bug. It will be great if you can submit a
> patch.
>
> Prasad
>
>
>
> From: David Lerman <dl...@videoegg.com>
> Reply-To: <hi...@hadoop.apache.org>
> Date: Fri, 4 Sep 2009 13:19:28 -0700
> To: <hi...@hadoop.apache.org>
> Subject: MetaStoreClient not getting cached
>
> Planning a query is taking a very long time, and during the pause, I see the
> following log lines hundreds of times in a row
>
> 2009-09-04 16:14:17,090 INFO metastore.HiveMetaStore
> (HiveMetaStore.java:getMS(125)) - 0: Opening raw store with implemenation
> class:org.apache.hadoop.hive.metastore.ObjectStore
> 2009-09-04 16:14:17,090
> INFO metastore.ObjectStore (ObjectStore.java:initialize(120)) -
> ObjectStore, initialize called
> 2009-09-04 16:14:17,090 INFO
> metastore.ObjectStore (ObjectStore.java:setConf(106)) - Initialized
> ObjectStore
>
> Going through the code, I see that the PartitionPruner is asking for
> metadata on every partition, and every call ends up calling
> org.apache.hadoop.hive.ql.metadata.Hive.getMSC():
>
> private IMetaStoreClient getMSC() throws MetaException {
> IMetaStoreClient msc = threadLocalMSC.get();
> if(msc == null) {
> msc = this.createMetaStoreClient();
> }
> return msc;
> }
>
> Each call ends up creating a new instance, because threadLocalMSC.set() is
> never called. Is there a reason this isn't:
>
> private IMetaStoreClient getMSC() throws MetaException {
> IMetaStoreClient msc = threadLocalMSC.get();
> if(msc == null) {
> msc = this.createMetaStoreClient();
> threadLocalMSC.set(msc); // CHANGE HERE!
> }
> return msc;
> }
>
> Thanks!
Re: MetaStoreClient not getting cached
Posted by Prasad Chakka <pc...@facebook.com>.
Yeah, you are correct. It is a bug. It will be great if you can submit a patch.
Prasad
________________________________
From: David Lerman <dl...@videoegg.com>
Reply-To: <hi...@hadoop.apache.org>
Date: Fri, 4 Sep 2009 13:19:28 -0700
To: <hi...@hadoop.apache.org>
Subject: MetaStoreClient not getting cached
Planning a query is taking a very long time, and during the pause, I see the
following log lines hundreds of times in a row
2009-09-04 16:14:17,090 INFO metastore.HiveMetaStore
(HiveMetaStore.java:getMS(125)) - 0: Opening raw store with implemenation
class:org.apache.hadoop.hive.metastore.ObjectStore
2009-09-04 16:14:17,090
INFO metastore.ObjectStore (ObjectStore.java:initialize(120)) -
ObjectStore, initialize called
2009-09-04 16:14:17,090 INFO
metastore.ObjectStore (ObjectStore.java:setConf(106)) - Initialized
ObjectStore
Going through the code, I see that the PartitionPruner is asking for
metadata on every partition, and every call ends up calling
org.apache.hadoop.hive.ql.metadata.Hive.getMSC():
private IMetaStoreClient getMSC() throws MetaException {
IMetaStoreClient msc = threadLocalMSC.get();
if(msc == null) {
msc = this.createMetaStoreClient();
}
return msc;
}
Each call ends up creating a new instance, because threadLocalMSC.set() is
never called. Is there a reason this isn't:
private IMetaStoreClient getMSC() throws MetaException {
IMetaStoreClient msc = threadLocalMSC.get();
if(msc == null) {
msc = this.createMetaStoreClient();
threadLocalMSC.set(msc); // CHANGE HERE!
}
return msc;
}
Thanks!
On 9/4/09 3:54 PM, "Vijay" <te...@gmail.com> wrote:
> Just a follow up to that: is that trunk only or does the 0.4 branch support
> hadoop-0.20.0 as well?
>
> On Fri, Sep 4, 2009 at 10:13 AM, Zheng Shao <zs...@gmail.com> wrote:
>> Yes.
>>
>> Zheng
>>
>>
>> On Fri, Sep 4, 2009 at 2:17 AM, Schubert Zhang <zs...@gmail.com> wrote:
>>>
>>> Does current Hive trunk support hadoop-0.20.0?
>>
>>
>>
>> --
>> Yours,
>> Zheng
>
MetaStoreClient not getting cached
Posted by David Lerman <dl...@videoegg.com>.
Planning a query is taking a very long time, and during the pause, I see the
following log lines hundreds of times in a row
2009-09-04 16:14:17,090 INFO metastore.HiveMetaStore
(HiveMetaStore.java:getMS(125)) - 0: Opening raw store with implemenation
class:org.apache.hadoop.hive.metastore.ObjectStore
2009-09-04 16:14:17,090
INFO metastore.ObjectStore (ObjectStore.java:initialize(120)) -
ObjectStore, initialize called
2009-09-04 16:14:17,090 INFO
metastore.ObjectStore (ObjectStore.java:setConf(106)) - Initialized
ObjectStore
Going through the code, I see that the PartitionPruner is asking for
metadata on every partition, and every call ends up calling
org.apache.hadoop.hive.ql.metadata.Hive.getMSC():
private IMetaStoreClient getMSC() throws MetaException {
IMetaStoreClient msc = threadLocalMSC.get();
if(msc == null) {
msc = this.createMetaStoreClient();
}
return msc;
}
Each call ends up creating a new instance, because threadLocalMSC.set() is
never called. Is there a reason this isn't:
private IMetaStoreClient getMSC() throws MetaException {
IMetaStoreClient msc = threadLocalMSC.get();
if(msc == null) {
msc = this.createMetaStoreClient();
threadLocalMSC.set(msc); // CHANGE HERE!
}
return msc;
}
Thanks!
On 9/4/09 3:54 PM, "Vijay" <te...@gmail.com> wrote:
> Just a follow up to that: is that trunk only or does the 0.4 branch support
> hadoop-0.20.0 as well?
>
> On Fri, Sep 4, 2009 at 10:13 AM, Zheng Shao <zs...@gmail.com> wrote:
>> Yes.
>>
>> Zheng
>>
>>
>> On Fri, Sep 4, 2009 at 2:17 AM, Schubert Zhang <zs...@gmail.com> wrote:
>>>
>>> Does current Hive trunk support hadoop-0.20.0?
>>
>>
>>
>> --
>> Yours,
>> Zheng
>
Re: Does current Hive trunk support hadoop-0.20.0?
Posted by Vijay <te...@gmail.com>.
Just a follow up to that: is that trunk only or does the 0.4 branch support
hadoop-0.20.0 as well?
On Fri, Sep 4, 2009 at 10:13 AM, Zheng Shao <zs...@gmail.com> wrote:
> Yes.
>
> Zheng
>
> On Fri, Sep 4, 2009 at 2:17 AM, Schubert Zhang <zs...@gmail.com> wrote:
>
>>
>> Does current Hive trunk support hadoop-0.20.0?
>>
>
>
>
> --
> Yours,
> Zheng
>
Re: Does current Hive trunk support hadoop-0.20.0?
Posted by Zheng Shao <zs...@gmail.com>.
Yes.
Zheng
On Fri, Sep 4, 2009 at 2:17 AM, Schubert Zhang <zs...@gmail.com> wrote:
>
> Does current Hive trunk support hadoop-0.20.0?
>
--
Yours,
Zheng