You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Min Zhou <co...@gmail.com> on 2009/05/09 18:00:20 UTC
Is it possible hiveserver both be a server and a client of itself?
Hi guys,
I found that HiveServer can fetch metadata thourgh Hive class after
generated a task like DDLTask, and meanwhile Hive class is a client
who ask HiveServer for metadata. So I guess it may be possible
hiveserver both be a server and a client of itself, am I right?
Thanks,
Min
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
RE: Is it possible hiveserver both be a server and a client of
itself?
Posted by Ashish Thusoo <at...@facebook.com>.
I see your point Min. Yes this is possible however only the metadata calls to the HiveServer should emanate from Hive.java. An execute should not call an execute etc... but it is possible in deployments where the HiveServer is its own clients and if I am correct thrift stubs should be able to deal with this and not make a network connection.
However, the ideal deployment would have a HiveServer in front of a metastore server (which is different). A cluster of HiveServers front a single metastore server. We would want the clustering on HiveServers as they serve out the data to clients talking jdbc. So in that deployment (which we think would make sense if a deployment is large scale and hit by many users), the metadata calls made though Hive.java (in turn called though the execute call in HiveServer) would just go to a different metastore server.
Ashish
________________________________________
From: Min Zhou [coderplay@gmail.com]
Sent: Wednesday, May 13, 2009 8:21 PM
To: hive-user@hadoop.apache.org
Subject: Re: Is it possible hiveserver both be a server and a client of itself?
HiveServer does call methods in Hive.java when compiling and planing a query command. HiveServer talk to itself through Hive.java and a socket connection.
This is the logic of the current hive trunk code.
On Thu, May 14, 2009 at 11:16 AM, Raghu Murthy <rm...@facebook.com>> wrote:
Hive.java does not call methods in HiveServer. Can you explain what you are
trying to achieve? You should just be able to start the HiveServer and then
create a HiveClient to invoke methods of both HiveServer itself and
HiveMetaStore.
On 5/13/09 8:07 PM, "Min Zhou" <co...@gmail.com>> wrote:
> Hey Raghu,
>
> Agreed with you. My question is , Hive.java code called by HiveServer, and
> Hive.java would talked to HiveServer through a socket connection. Am I right?
>
> Min
> On Thu, May 14, 2009 at 11:02 AM, Raghu Murthy <rm...@facebook.com>> wrote:
>> We have used thrift's inheritance feature where one service can extend
>> another service, i.e., HiveServer extends HiveMetaStore. This means that
>> 1. HiveServer exposes HiveMetaStore methods
>> 2. HiveClient can be used to invoke methods exposed by both HiveServer and
>> HiveMetaStore.
>>
>> Does that make sense?
>>
>>
>> On 5/13/09 7:52 PM, "Min Zhou" <co...@gmail.com>> wrote:
>>
>>>> Sorry, I didn't precisely express my meaning. I didn't talk about a
>>>> supposition, but talked about a possible phenomenon may happen at hive.
>>>> The hiveserver may both be a server and a client of itself, you can look
>>>> >> at
>>>> the definition of HiveServer
>>>>
>>>> public static class HiveServerHandler extends HiveMetaStore.HMSHandler ...
>>>> {
>>>> ...
>>>> }
>>>>
>>>>
>>>>
>>>> It extends HiveMetaStore.HMSHandler, which provide the same function of
>>>> metastore .
>>>>
>>>> public class HiveMetaStore extends ThriftHiveMetastore {
>>>>
>>>> public static class HMSHandler extends FacebookBase implements
>>>> ThriftHiveMetastore.Iface {
>>>> // omit
>>>> }
>>>>
>>>> // main entry of the MetaStore but didn't be called at this case
>>>> public static void main(String[] args) {
>>>> ...
>>>> }
>>>> }
>>>>
>>>> there is no another standalone metastore server if you have up a
>>> hiveserver,
>>>> which is also a metaserver itself.
>>>>
>>>> Min
>>>> On Thu, May 14, 2009 at 10:35 AM, Amr Awadallah <aa...@cloudera.com>> wrote:
>>>>>>
>>>>>>
>>>>>> I don't think that would work, where would you store the meta-data for
>>>>>> >>> the
>>>>>> meta-data?
>>>>>>
>>>>>>
>>>>>> Min Zhou wrote:
>>>>>>>> Hi Ashish,
>>>>>>>> Thank you for your swift reply. I guess it's HiveServer code inherit
>>>>>>>> the
>>>>>>>> metastore thrift api. HiveServer provide metadata service by itself,
>>>>>>>> not
>>>>>>>> call another metastore server, and Hive.class talks to the HiveServer
>>>>>>>> to
>>>>>>>> get
>>>>>>>> the metadata.
>>>>>>>> Regards,
>>>>>>>> Min
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <at...@facebook.com>>
>>>>> wrote:
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Apologies for the delay Min..
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The HiveServer code calls the metastore thrift api to get the
>>>>>> metadata. The
>>>>>>>>>> Hive.class in ql also talks to the metastore api to get the metadata.
>>>>>>>>>> Not
>>>>>>>>>> sure if I follow your question though?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ashish
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> From: Min Zhou [mailto:coderplay@gmail.com<ma...@gmail.com>]
>>>>>>>>>> Sent: Wednesday, May 13, 2009 6:07 PM
>>>>>>>>>> To: hive-user
>>>>>>>>>> Subject: Re: Is it possible hiveserver both be a server and a client
>>>>>>>>>> of
>>>>>>>>>> itself?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Is there anyone can answer my question?
>>>>>>>>>> Thanks in advanced!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, May 10, 2009 at 12:00 AM, Min Zhou <co...@gmail.com>>
>>>>>>>>>> >>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>
>>>>>>>>>>>> I found that HiveServer can fetch metadata thourgh Hive class after
>>>>>>>>>>>> generated a task like DDLTask, and meanwhile Hive class is a client
>>>>>>>>>>>> who ask HiveServer for metadata. So I guess it may be possible
>>>>>>>>>>>> hiveserver both be a server and a client of itself, am I right?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Min
>>>>>>>>>>>> --
>>>>>>>>>>>> My research interests are distributed systems, parallel computing
>>>>>>>>>>>> and
>>>>>>>>>>>> bytecode based virtual machine.
>>>>>>>>>>>>
>>>>>>>>>>>> My profile:
>>>>>>>>>>>> http://www.linkedin.com/in/coderplay
>>>>>>>>>>>> My blog:
>>>>>>>>>>>> http://coderplay.javaeye.com
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> My research interests are distributed systems, parallel computing and
>>>>>>>>>> bytecode based virtual machine.
>>>>>>>>>>
>>>>>>>>>> My profile:
>>>>>>>>>> http://www.linkedin.com/in/coderplay
>>>>>>>>>> My blog:
>>>>>>>>>> http://coderplay.javaeye.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> My research interests are distributed systems, parallel computing and
>>>>>>>> bytecode based virtual machine.
>>>>>>>>
>>>>>>>> My profile:
>>>>>>>> http://www.linkedin.com/in/coderplay
>>>>>>>> My blog:
>>>>>>>> http://coderplay.javaeye.com
>>>>
>>>>
>>
>
>
--
My research interests are distributed systems, parallel computing and bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
Re: Is it possible hiveserver both be a server and a client of
itself?
Posted by Min Zhou <co...@gmail.com>.
HiveServer does call methods in Hive.java when compiling and planing a query
command. HiveServer talk to itself through Hive.java and a socket
connection.
This is the logic of the current hive trunk code.
On Thu, May 14, 2009 at 11:16 AM, Raghu Murthy <rm...@facebook.com> wrote:
> Hive.java does not call methods in HiveServer. Can you explain what you are
> trying to achieve? You should just be able to start the HiveServer and then
> create a HiveClient to invoke methods of both HiveServer itself and
> HiveMetaStore.
>
> On 5/13/09 8:07 PM, "Min Zhou" <co...@gmail.com> wrote:
>
> > Hey Raghu,
> >
> > Agreed with you. My question is , Hive.java code called by HiveServer,
> and
> > Hive.java would talked to HiveServer through a socket connection. Am I
> right?
> >
> > Min
> > On Thu, May 14, 2009 at 11:02 AM, Raghu Murthy <rm...@facebook.com>
> wrote:
> >> We have used thrift's inheritance feature where one service can extend
> >> another service, i.e., HiveServer extends HiveMetaStore. This means that
> >> 1. HiveServer exposes HiveMetaStore methods
> >> 2. HiveClient can be used to invoke methods exposed by both HiveServer
> and
> >> HiveMetaStore.
> >>
> >> Does that make sense?
> >>
> >>
> >> On 5/13/09 7:52 PM, "Min Zhou" <co...@gmail.com> wrote:
> >>
> >>>> Sorry, I didn't precisely express my meaning. I didn't talk about a
> >>>> supposition, but talked about a possible phenomenon may happen at
> hive.
> >>>> The hiveserver may both be a server and a client of itself, you can
> look
> >>>> >> at
> >>>> the definition of HiveServer
> >>>>
> >>>> public static class HiveServerHandler extends HiveMetaStore.HMSHandler
> ...
> >>>> {
> >>>> ...
> >>>> }
> >>>>
> >>>>
> >>>>
> >>>> It extends HiveMetaStore.HMSHandler, which provide the same function
> of
> >>>> metastore .
> >>>>
> >>>> public class HiveMetaStore extends ThriftHiveMetastore {
> >>>>
> >>>> public static class HMSHandler extends FacebookBase implements
> >>>> ThriftHiveMetastore.Iface {
> >>>> // omit
> >>>> }
> >>>>
> >>>> // main entry of the MetaStore but didn't be called at this case
> >>>> public static void main(String[] args) {
> >>>> ...
> >>>> }
> >>>> }
> >>>>
> >>>> there is no another standalone metastore server if you have up a
> >>> hiveserver,
> >>>> which is also a metaserver itself.
> >>>>
> >>>> Min
> >>>> On Thu, May 14, 2009 at 10:35 AM, Amr Awadallah <aa...@cloudera.com>
> wrote:
> >>>>>>
> >>>>>>
> >>>>>> I don't think that would work, where would you store the meta-data
> for
> >>>>>> >>> the
> >>>>>> meta-data?
> >>>>>>
> >>>>>>
> >>>>>> Min Zhou wrote:
> >>>>>>>> Hi Ashish,
> >>>>>>>> Thank you for your swift reply. I guess it's HiveServer code
> inherit
> >>>>>>>> the
> >>>>>>>> metastore thrift api. HiveServer provide metadata service by
> itself,
> >>>>>>>> not
> >>>>>>>> call another metastore server, and Hive.class talks to the
> HiveServer
> >>>>>>>> to
> >>>>>>>> get
> >>>>>>>> the metadata.
> >>>>>>>> Regards,
> >>>>>>>> Min
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <
> athusoo@facebook.com>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Apologies for the delay Min..
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> The HiveServer code calls the metastore thrift api to get the
> >>>>>> metadata. The
> >>>>>>>>>> Hive.class in ql also talks to the metastore api to get the
> metadata.
> >>>>>>>>>> Not
> >>>>>>>>>> sure if I follow your question though?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Ashish
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> From: Min Zhou [mailto:coderplay@gmail.com]
> >>>>>>>>>> Sent: Wednesday, May 13, 2009 6:07 PM
> >>>>>>>>>> To: hive-user
> >>>>>>>>>> Subject: Re: Is it possible hiveserver both be a server and a
> client
> >>>>>>>>>> of
> >>>>>>>>>> itself?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Is there anyone can answer my question?
> >>>>>>>>>> Thanks in advanced!
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Sun, May 10, 2009 at 12:00 AM, Min Zhou <coderplay@gmail.com
> >
> >>>>>>>>>> >>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>> Hi guys,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I found that HiveServer can fetch metadata thourgh Hive class
> after
> >>>>>>>>>>>> generated a task like DDLTask, and meanwhile Hive class is a
> client
> >>>>>>>>>>>> who ask HiveServer for metadata. So I guess it may be possible
> >>>>>>>>>>>> hiveserver both be a server and a client of itself, am I
> right?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Min
> >>>>>>>>>>>> --
> >>>>>>>>>>>> My research interests are distributed systems, parallel
> computing
> >>>>>>>>>>>> and
> >>>>>>>>>>>> bytecode based virtual machine.
> >>>>>>>>>>>>
> >>>>>>>>>>>> My profile:
> >>>>>>>>>>>> http://www.linkedin.com/in/coderplay
> >>>>>>>>>>>> My blog:
> >>>>>>>>>>>> http://coderplay.javaeye.com
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> My research interests are distributed systems, parallel
> computing and
> >>>>>>>>>> bytecode based virtual machine.
> >>>>>>>>>>
> >>>>>>>>>> My profile:
> >>>>>>>>>> http://www.linkedin.com/in/coderplay
> >>>>>>>>>> My blog:
> >>>>>>>>>> http://coderplay.javaeye.com
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> My research interests are distributed systems, parallel computing
> and
> >>>>>>>> bytecode based virtual machine.
> >>>>>>>>
> >>>>>>>> My profile:
> >>>>>>>> http://www.linkedin.com/in/coderplay
> >>>>>>>> My blog:
> >>>>>>>> http://coderplay.javaeye.com
> >>>>
> >>>>
> >>
> >
> >
>
>
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
Re: Is it possible hiveserver both be a server and a client of
itself?
Posted by Raghu Murthy <rm...@facebook.com>.
Hive.java does not call methods in HiveServer. Can you explain what you are
trying to achieve? You should just be able to start the HiveServer and then
create a HiveClient to invoke methods of both HiveServer itself and
HiveMetaStore.
On 5/13/09 8:07 PM, "Min Zhou" <co...@gmail.com> wrote:
> Hey Raghu,
>
> Agreed with you. My question is , Hive.java code called by HiveServer, and
> Hive.java would talked to HiveServer through a socket connection. Am I right?
>
> Min
> On Thu, May 14, 2009 at 11:02 AM, Raghu Murthy <rm...@facebook.com> wrote:
>> We have used thrift's inheritance feature where one service can extend
>> another service, i.e., HiveServer extends HiveMetaStore. This means that
>> 1. HiveServer exposes HiveMetaStore methods
>> 2. HiveClient can be used to invoke methods exposed by both HiveServer and
>> HiveMetaStore.
>>
>> Does that make sense?
>>
>>
>> On 5/13/09 7:52 PM, "Min Zhou" <co...@gmail.com> wrote:
>>
>>>> Sorry, I didn't precisely express my meaning. I didn't talk about a
>>>> supposition, but talked about a possible phenomenon may happen at hive.
>>>> The hiveserver may both be a server and a client of itself, you can look
>>>> >> at
>>>> the definition of HiveServer
>>>>
>>>> public static class HiveServerHandler extends HiveMetaStore.HMSHandler ...
>>>> {
>>>> ...
>>>> }
>>>>
>>>>
>>>>
>>>> It extends HiveMetaStore.HMSHandler, which provide the same function of
>>>> metastore .
>>>>
>>>> public class HiveMetaStore extends ThriftHiveMetastore {
>>>>
>>>> public static class HMSHandler extends FacebookBase implements
>>>> ThriftHiveMetastore.Iface {
>>>> // omit
>>>> }
>>>>
>>>> // main entry of the MetaStore but didn't be called at this case
>>>> public static void main(String[] args) {
>>>> ...
>>>> }
>>>> }
>>>>
>>>> there is no another standalone metastore server if you have up a
>>> hiveserver,
>>>> which is also a metaserver itself.
>>>>
>>>> Min
>>>> On Thu, May 14, 2009 at 10:35 AM, Amr Awadallah <aa...@cloudera.com> wrote:
>>>>>>
>>>>>>
>>>>>> I don't think that would work, where would you store the meta-data for
>>>>>> >>> the
>>>>>> meta-data?
>>>>>>
>>>>>>
>>>>>> Min Zhou wrote:
>>>>>>>> Hi Ashish,
>>>>>>>> Thank you for your swift reply. I guess it's HiveServer code inherit
>>>>>>>> the
>>>>>>>> metastore thrift api. HiveServer provide metadata service by itself,
>>>>>>>> not
>>>>>>>> call another metastore server, and Hive.class talks to the HiveServer
>>>>>>>> to
>>>>>>>> get
>>>>>>>> the metadata.
>>>>>>>> Regards,
>>>>>>>> Min
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <at...@facebook.com>
>>>>> wrote:
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Apologies for the delay Min..
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The HiveServer code calls the metastore thrift api to get the
>>>>>> metadata. The
>>>>>>>>>> Hive.class in ql also talks to the metastore api to get the metadata.
>>>>>>>>>> Not
>>>>>>>>>> sure if I follow your question though?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ashish
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> From: Min Zhou [mailto:coderplay@gmail.com]
>>>>>>>>>> Sent: Wednesday, May 13, 2009 6:07 PM
>>>>>>>>>> To: hive-user
>>>>>>>>>> Subject: Re: Is it possible hiveserver both be a server and a client
>>>>>>>>>> of
>>>>>>>>>> itself?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Is there anyone can answer my question?
>>>>>>>>>> Thanks in advanced!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, May 10, 2009 at 12:00 AM, Min Zhou <co...@gmail.com>
>>>>>>>>>> >>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>
>>>>>>>>>>>> I found that HiveServer can fetch metadata thourgh Hive class after
>>>>>>>>>>>> generated a task like DDLTask, and meanwhile Hive class is a client
>>>>>>>>>>>> who ask HiveServer for metadata. So I guess it may be possible
>>>>>>>>>>>> hiveserver both be a server and a client of itself, am I right?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Min
>>>>>>>>>>>> --
>>>>>>>>>>>> My research interests are distributed systems, parallel computing
>>>>>>>>>>>> and
>>>>>>>>>>>> bytecode based virtual machine.
>>>>>>>>>>>>
>>>>>>>>>>>> My profile:
>>>>>>>>>>>> http://www.linkedin.com/in/coderplay
>>>>>>>>>>>> My blog:
>>>>>>>>>>>> http://coderplay.javaeye.com
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> My research interests are distributed systems, parallel computing and
>>>>>>>>>> bytecode based virtual machine.
>>>>>>>>>>
>>>>>>>>>> My profile:
>>>>>>>>>> http://www.linkedin.com/in/coderplay
>>>>>>>>>> My blog:
>>>>>>>>>> http://coderplay.javaeye.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> My research interests are distributed systems, parallel computing and
>>>>>>>> bytecode based virtual machine.
>>>>>>>>
>>>>>>>> My profile:
>>>>>>>> http://www.linkedin.com/in/coderplay
>>>>>>>> My blog:
>>>>>>>> http://coderplay.javaeye.com
>>>>
>>>>
>>
>
>
Re: Is it possible hiveserver both be a server and a client of
itself?
Posted by Min Zhou <co...@gmail.com>.
Hey Raghu,
Agreed with you. My question is , Hive.java code called by HiveServer, and
Hive.java would talked to HiveServer through a socket connection. Am I
right?
Min
On Thu, May 14, 2009 at 11:02 AM, Raghu Murthy <rm...@facebook.com> wrote:
> We have used thrift's inheritance feature where one service can extend
> another service, i.e., HiveServer extends HiveMetaStore. This means that
> 1. HiveServer exposes HiveMetaStore methods
> 2. HiveClient can be used to invoke methods exposed by both HiveServer and
> HiveMetaStore.
>
> Does that make sense?
>
>
> On 5/13/09 7:52 PM, "Min Zhou" <co...@gmail.com> wrote:
>
> > Sorry, I didn't precisely express my meaning. I didn't talk about a
> > supposition, but talked about a possible phenomenon may happen at hive.
> > The hiveserver may both be a server and a client of itself, you can look
> at
> > the definition of HiveServer
> >
> > public static class HiveServerHandler extends HiveMetaStore.HMSHandler
> ... {
> > ...
> > }
> >
> >
> >
> > It extends HiveMetaStore.HMSHandler, which provide the same function of
> > metastore .
> >
> > public class HiveMetaStore extends ThriftHiveMetastore {
> >
> > public static class HMSHandler extends FacebookBase implements
> > ThriftHiveMetastore.Iface {
> > // omit
> > }
> >
> > // main entry of the MetaStore but didn't be called at this case
> > public static void main(String[] args) {
> > ...
> > }
> > }
> >
> > there is no another standalone metastore server if you have up a
> hiveserver,
> > which is also a metaserver itself.
> >
> > Min
> > On Thu, May 14, 2009 at 10:35 AM, Amr Awadallah <aa...@cloudera.com>
> wrote:
> >>
> >>
> >> I don't think that would work, where would you store the meta-data for
> the
> >> meta-data?
> >>
> >>
> >> Min Zhou wrote:
> >>> Hi Ashish,
> >>> Thank you for your swift reply. I guess it's HiveServer code inherit
> the
> >>> metastore thrift api. HiveServer provide metadata service by itself,
> not
> >>> call another metastore server, and Hive.class talks to the HiveServer
> to
> >>> get
> >>> the metadata.
> >>> Regards,
> >>> Min
> >>>
> >>>
> >>> On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <at...@facebook.com>
> wrote:
> >>>
> >>>>
> >>>>
> >>>> Apologies for the delay Min..
> >>>>
> >>>>
> >>>>
> >>>> The HiveServer code calls the metastore thrift api to get the
> metadata. The
> >>>> Hive.class in ql also talks to the metastore api to get the metadata.
> Not
> >>>> sure if I follow your question though?
> >>>>
> >>>>
> >>>>
> >>>> Ashish
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> From: Min Zhou [mailto:coderplay@gmail.com]
> >>>> Sent: Wednesday, May 13, 2009 6:07 PM
> >>>> To: hive-user
> >>>> Subject: Re: Is it possible hiveserver both be a server and a client
> of
> >>>> itself?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Is there anyone can answer my question?
> >>>> Thanks in advanced!
> >>>>
> >>>>
> >>>> On Sun, May 10, 2009 at 12:00 AM, Min Zhou <co...@gmail.com>
> wrote:
> >>>>
> >>>>> Hi guys,
> >>>>>
> >>>>> I found that HiveServer can fetch metadata thourgh Hive class after
> >>>>> generated a task like DDLTask, and meanwhile Hive class is a client
> >>>>> who ask HiveServer for metadata. So I guess it may be possible
> >>>>> hiveserver both be a server and a client of itself, am I right?
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>> Min
> >>>>> --
> >>>>> My research interests are distributed systems, parallel computing and
> >>>>> bytecode based virtual machine.
> >>>>>
> >>>>> My profile:
> >>>>> http://www.linkedin.com/in/coderplay
> >>>>> My blog:
> >>>>> http://coderplay.javaeye.com
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> My research interests are distributed systems, parallel computing and
> >>>> bytecode based virtual machine.
> >>>>
> >>>> My profile:
> >>>> http://www.linkedin.com/in/coderplay
> >>>> My blog:
> >>>> http://coderplay.javaeye.com
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> My research interests are distributed systems, parallel computing and
> >>> bytecode based virtual machine.
> >>>
> >>> My profile:
> >>> http://www.linkedin.com/in/coderplay
> >>> My blog:
> >>> http://coderplay.javaeye.com
> >
> >
>
>
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
Re: Is it possible hiveserver both be a server and a client of
itself?
Posted by Raghu Murthy <rm...@facebook.com>.
We have used thrift's inheritance feature where one service can extend
another service, i.e., HiveServer extends HiveMetaStore. This means that
1. HiveServer exposes HiveMetaStore methods
2. HiveClient can be used to invoke methods exposed by both HiveServer and
HiveMetaStore.
Does that make sense?
On 5/13/09 7:52 PM, "Min Zhou" <co...@gmail.com> wrote:
> Sorry, I didn't precisely express my meaning. I didn't talk about a
> supposition, but talked about a possible phenomenon may happen at hive.
> The hiveserver may both be a server and a client of itself, you can look at
> the definition of HiveServer
>
> public static class HiveServerHandler extends HiveMetaStore.HMSHandler ... {
> ...
> }
>
>
>
> It extends HiveMetaStore.HMSHandler, which provide the same function of
> metastore .
>
> public class HiveMetaStore extends ThriftHiveMetastore {
>
> public static class HMSHandler extends FacebookBase implements
> ThriftHiveMetastore.Iface {
> // omit
> }
>
> // main entry of the MetaStore but didn't be called at this case
> public static void main(String[] args) {
> ...
> }
> }
>
> there is no another standalone metastore server if you have up a hiveserver,
> which is also a metaserver itself.
>
> Min
> On Thu, May 14, 2009 at 10:35 AM, Amr Awadallah <aa...@cloudera.com> wrote:
>>
>>
>> I don't think that would work, where would you store the meta-data for the
>> meta-data?
>>
>>
>> Min Zhou wrote:
>>> Hi Ashish,
>>> Thank you for your swift reply. I guess it's HiveServer code inherit the
>>> metastore thrift api. HiveServer provide metadata service by itself, not
>>> call another metastore server, and Hive.class talks to the HiveServer to
>>> get
>>> the metadata.
>>> Regards,
>>> Min
>>>
>>>
>>> On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <at...@facebook.com> wrote:
>>>
>>>>
>>>>
>>>> Apologies for the delay Min..
>>>>
>>>>
>>>>
>>>> The HiveServer code calls the metastore thrift api to get the metadata. The
>>>> Hive.class in ql also talks to the metastore api to get the metadata. Not
>>>> sure if I follow your question though?
>>>>
>>>>
>>>>
>>>> Ashish
>>>>
>>>>
>>>>
>>>>
>>>> From: Min Zhou [mailto:coderplay@gmail.com]
>>>> Sent: Wednesday, May 13, 2009 6:07 PM
>>>> To: hive-user
>>>> Subject: Re: Is it possible hiveserver both be a server and a client of
>>>> itself?
>>>>
>>>>
>>>>
>>>>
>>>> Is there anyone can answer my question?
>>>> Thanks in advanced!
>>>>
>>>>
>>>> On Sun, May 10, 2009 at 12:00 AM, Min Zhou <co...@gmail.com> wrote:
>>>>
>>>>> Hi guys,
>>>>>
>>>>> I found that HiveServer can fetch metadata thourgh Hive class after
>>>>> generated a task like DDLTask, and meanwhile Hive class is a client
>>>>> who ask HiveServer for metadata. So I guess it may be possible
>>>>> hiveserver both be a server and a client of itself, am I right?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Min
>>>>> --
>>>>> My research interests are distributed systems, parallel computing and
>>>>> bytecode based virtual machine.
>>>>>
>>>>> My profile:
>>>>> http://www.linkedin.com/in/coderplay
>>>>> My blog:
>>>>> http://coderplay.javaeye.com
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> My research interests are distributed systems, parallel computing and
>>>> bytecode based virtual machine.
>>>>
>>>> My profile:
>>>> http://www.linkedin.com/in/coderplay
>>>> My blog:
>>>> http://coderplay.javaeye.com
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> My research interests are distributed systems, parallel computing and
>>> bytecode based virtual machine.
>>>
>>> My profile:
>>> http://www.linkedin.com/in/coderplay
>>> My blog:
>>> http://coderplay.javaeye.com
>
>
Re: Is it possible hiveserver both be a server and a client of
itself?
Posted by Min Zhou <co...@gmail.com>.
Sorry, I didn't precisely express my meaning. I didn't talk about a
supposition, but talked about a possible phenomenon may happen at hive.
The hiveserver may both be a server and a client of itself, you can look at
the definition of HiveServer
public static class HiveServerHandler extends HiveMetaStore.HMSHandler ... {
...
}
It extends HiveMetaStore.HMSHandler, which provide the same function of
metastore .
public class HiveMetaStore extends ThriftHiveMetastore {
public static class HMSHandler extends FacebookBase implements
ThriftHiveMetastore.Iface {
// omit
}
// main entry of the MetaStore but didn't be called at this case
public static void main(String[] args) {
...
}
}
there is no another standalone metastore server if you have up a hiveserver,
which is also a metaserver itself.
Min
On Thu, May 14, 2009 at 10:35 AM, Amr Awadallah <aa...@cloudera.com> wrote:
>
> I don't think that would work, where would you store the meta-data for the
> meta-data?
>
>
> Min Zhou wrote:
>
> Hi Ashish,
> Thank you for your swift reply. I guess it's HiveServer code inherit the
> metastore thrift api. HiveServer provide metadata service by itself, not
> call another metastore server, and Hive.class talks to the HiveServer to
> get the metadata.
> Regards,
> Min
>
> On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <at...@facebook.com>wrote:
>
>> Apologies for the delay Min..
>>
>> The HiveServer code calls the metastore thrift api to get the metadata.
>> The Hive.class in ql also talks to the metastore api to get the metadata.
>> Not sure if I follow your question though?
>>
>> Ashish
>>
>> ------------------------------
>> *From:* Min Zhou [mailto:coderplay@gmail.com]
>> *Sent:* Wednesday, May 13, 2009 6:07 PM
>> *To:* hive-user
>> *Subject:* Re: Is it possible hiveserver both be a server and a client of
>> itself?
>>
>> Is there anyone can answer my question?
>> Thanks in advanced!
>>
>> On Sun, May 10, 2009 at 12:00 AM, Min Zhou <co...@gmail.com> wrote:
>>
>>> Hi guys,
>>>
>>> I found that HiveServer can fetch metadata thourgh Hive class after
>>> generated a task like DDLTask, and meanwhile Hive class is a client
>>> who ask HiveServer for metadata. So I guess it may be possible
>>> hiveserver both be a server and a client of itself, am I right?
>>>
>>>
>>> Thanks,
>>> Min
>>> --
>>> My research interests are distributed systems, parallel computing and
>>> bytecode based virtual machine.
>>>
>>> My profile:
>>> http://www.linkedin.com/in/coderplay
>>> My blog:
>>> http://coderplay.javaeye.com
>>>
>>
>>
>>
>> --
>> My research interests are distributed systems, parallel computing and
>> bytecode based virtual machine.
>>
>> My profile:
>> http://www.linkedin.com/in/coderplay
>> My blog:
>> http://coderplay.javaeye.com
>>
>
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>
>
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
Re: Is it possible hiveserver both be a server and a client of
itself?
Posted by Prasad Chakka <pc...@facebook.com>.
HiveServer is also a HiveMetastoreServer in usual configuration. But I think it is possible make HiveServer talk to a different server for metadata by changing hive-site.xml that HiveServer loads. I am not sure why it would be useful though. Though I didn't try this configuration.
Prasad
________________________________
From: Amr Awadallah <aa...@cloudera.com>
Reply-To: <hi...@hadoop.apache.org>
Date: Wed, 13 May 2009 19:35:40 -0700
To: <hi...@hadoop.apache.org>
Subject: Re: Is it possible hiveserver both be a server and a client of itself?
I don't think that would work, where would you store the meta-data for the meta-data?
Min Zhou wrote:
Hi Ashish,
Thank you for your swift reply. I guess it's HiveServer code inherit the metastore thrift api. HiveServer provide metadata service by itself, not call another metastore server, and Hive.class talks to the HiveServer to get the metadata.
Regards,
Min
On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <at...@facebook.com> wrote:
Apologies for the delay Min..
The HiveServer code calls the metastore thrift api to get the metadata. The Hive.class in ql also talks to the metastore api to get the metadata. Not sure if I follow your question though?
Ashish
________________________________
From: Min Zhou [mailto:coderplay@gmail.com]
Sent: Wednesday, May 13, 2009 6:07 PM
To: hive-user
Subject: Re: Is it possible hiveserver both be a server and a client of itself?
Is there anyone can answer my question?
Thanks in advanced!
On Sun, May 10, 2009 at 12:00 AM, Min Zhou <co...@gmail.com> wrote:
Hi guys,
I found that HiveServer can fetch metadata thourgh Hive class after
generated a task like DDLTask, and meanwhile Hive class is a client
who ask HiveServer for metadata. So I guess it may be possible
hiveserver both be a server and a client of itself, am I right?
Thanks,
Min
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
--
My research interests are distributed systems, parallel computing and bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
--
My research interests are distributed systems, parallel computing and bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
Re: Is it possible hiveserver both be a server and a client of itself?
Posted by Amr Awadallah <aa...@cloudera.com>.
I don't think that would work, where would you store the meta-data for
the meta-data?
Min Zhou wrote:
> Hi Ashish,
> Thank you for your swift reply. I guess it's HiveServer code inherit
> the metastore thrift api. HiveServer provide metadata service by
> itself, not call another metastore server, and Hive.class talks to
> the HiveServer to get the metadata.
> Regards,
> Min
>
> On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <athusoo@facebook.com
> <ma...@facebook.com>> wrote:
>
> Apologies for the delay Min..
>
> The HiveServer code calls the metastore thrift api to get the
> metadata. The Hive.class in ql also talks to the metastore api to
> get the metadata. Not sure if I follow your question though?
>
> Ashish
>
> ------------------------------------------------------------------------
> *From:* Min Zhou [mailto:coderplay@gmail.com
> <ma...@gmail.com>]
> *Sent:* Wednesday, May 13, 2009 6:07 PM
> *To:* hive-user
> *Subject:* Re: Is it possible hiveserver both be a server and a
> client of itself?
>
> Is there anyone can answer my question?
> Thanks in advanced!
>
> On Sun, May 10, 2009 at 12:00 AM, Min Zhou <coderplay@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi guys,
>
> I found that HiveServer can fetch metadata thourgh Hive class
> after
> generated a task like DDLTask, and meanwhile Hive class is a
> client
> who ask HiveServer for metadata. So I guess it may be possible
> hiveserver both be a server and a client of itself, am I right?
>
>
> Thanks,
> Min
> --
> My research interests are distributed systems, parallel
> computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>
>
>
>
> --
> My research interests are distributed systems, parallel computing
> and bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>
>
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
Re: Is it possible hiveserver both be a server and a client of
itself?
Posted by Min Zhou <co...@gmail.com>.
Hi Ashish,
Thank you for your swift reply. I guess it's HiveServer code inherit the
metastore thrift api. HiveServer provide metadata service by itself, not
call another metastore server, and Hive.class talks to the HiveServer to
get the metadata.
Regards,
Min
On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <at...@facebook.com> wrote:
> Apologies for the delay Min..
>
> The HiveServer code calls the metastore thrift api to get the metadata. The
> Hive.class in ql also talks to the metastore api to get the metadata. Not
> sure if I follow your question though?
>
> Ashish
>
> ------------------------------
> *From:* Min Zhou [mailto:coderplay@gmail.com]
> *Sent:* Wednesday, May 13, 2009 6:07 PM
> *To:* hive-user
> *Subject:* Re: Is it possible hiveserver both be a server and a client of
> itself?
>
> Is there anyone can answer my question?
> Thanks in advanced!
>
> On Sun, May 10, 2009 at 12:00 AM, Min Zhou <co...@gmail.com> wrote:
>
>> Hi guys,
>>
>> I found that HiveServer can fetch metadata thourgh Hive class after
>> generated a task like DDLTask, and meanwhile Hive class is a client
>> who ask HiveServer for metadata. So I guess it may be possible
>> hiveserver both be a server and a client of itself, am I right?
>>
>>
>> Thanks,
>> Min
>> --
>> My research interests are distributed systems, parallel computing and
>> bytecode based virtual machine.
>>
>> My profile:
>> http://www.linkedin.com/in/coderplay
>> My blog:
>> http://coderplay.javaeye.com
>>
>
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
RE: Is it possible hiveserver both be a server and a client of
itself?
Posted by Ashish Thusoo <at...@facebook.com>.
Apologies for the delay Min..
The HiveServer code calls the metastore thrift api to get the metadata. The Hive.class in ql also talks to the metastore api to get the metadata. Not sure if I follow your question though?
Ashish
________________________________
From: Min Zhou [mailto:coderplay@gmail.com]
Sent: Wednesday, May 13, 2009 6:07 PM
To: hive-user
Subject: Re: Is it possible hiveserver both be a server and a client of itself?
Is there anyone can answer my question?
Thanks in advanced!
On Sun, May 10, 2009 at 12:00 AM, Min Zhou <co...@gmail.com>> wrote:
Hi guys,
I found that HiveServer can fetch metadata thourgh Hive class after
generated a task like DDLTask, and meanwhile Hive class is a client
who ask HiveServer for metadata. So I guess it may be possible
hiveserver both be a server and a client of itself, am I right?
Thanks,
Min
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
--
My research interests are distributed systems, parallel computing and bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
Re: Is it possible hiveserver both be a server and a client of
itself?
Posted by Min Zhou <co...@gmail.com>.
Is there anyone can answer my question?
Thanks in advanced!
On Sun, May 10, 2009 at 12:00 AM, Min Zhou <co...@gmail.com> wrote:
> Hi guys,
>
> I found that HiveServer can fetch metadata thourgh Hive class after
> generated a task like DDLTask, and meanwhile Hive class is a client
> who ask HiveServer for metadata. So I guess it may be possible
> hiveserver both be a server and a client of itself, am I right?
>
>
> Thanks,
> Min
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com