You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "V.Sriram" <sr...@ask.com> on 2011/11/01 08:28:16 UTC
Hbase Multiple Master issue
Hi all,
I am evaluating HBase Multiple master utility as I require that
funtionality. I used a simple 4 machine hbase cluster with M1 as master. Now
I started the Hmaster in M4 as well. I could see logs in M4 Master log as,
This HMaster is waiting as already M1 is acting as HMaster.
After this I stopped the Master in the M1 and after few seconds, the M4
master took over. But I am not seeing any region server communicating to
this new master. The region server logs show,
2011-10-31 23:21:18,834 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
master. Retrying. Error was:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy3.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at
org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1443)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586)
at java.lang.Thread.run(Thread.java:619)
Also in the M4 Master log I am seeing,
2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 60010
webServer.getConnectors()[0].getLocalPort() returned 60010
2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 60010
2011-10-31 23:20:47,671 INFO org.mortbay.log: jetty-6.1.26
2011-10-31 23:20:48,691 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:60010
2011-10-31 23:20:48,691 DEBUG org.apache.hadoop.hbase.master.HMaster:
Started service threads
2011-10-31 23:20:50,192 INFO org.apache.hadoop.hbase.master.ServerManager:
Waiting on regionserver(s) to checkin
2011-10-31 23:20:51,695 INFO org.apache.hadoop.hbase.master.ServerManager:
Waiting on regionserver(s) to checkin
2011-10-31 23:20:53,197 INFO org.apache.hadoop.hbase.master.ServerManager:
Waiting on regionserver(s) to checkin
Seems the region servers are only communicating to the M1 Hmaster and not to
the new one!!
Any help regarding this is highly appreciated :handshake:
--
View this message in context: http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32757197.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: How to import binary data using importTSV
Posted by Stack <st...@duboce.net>.
On Thu, Nov 3, 2011 at 3:29 PM, ChongQing Xiao <Cq...@epic.com> wrote:
> Does anyone know if ImportTsv supports importing binary data?
> From the source code, it seems it only support text in the .tsv file for the column value.
>
If I were to guess, a tool named importtsv probably doesn't do binary.
> It seems pretty simple to add encoding (say using BASE64) to support binary row key or binary data.
>
You could base64 encode and then use this in importtsv or, subclass or
replace the tsv part of the importtsv tool.
St.Ack
How to import binary data using importTSV
Posted by ChongQing Xiao <Cq...@epic.com>.
Hi,
Does anyone know if ImportTsv supports importing binary data?
>From the source code, it seems it only support text in the .tsv file for the column value.
It seems pretty simple to add encoding (say using BASE64) to support binary row key or binary data.
Thanks
Chong
RE: RE: Hbase Multiple Master issue
Posted by Ramkrishna S Vasudevan <ra...@huawei.com>.
But you say you cannot move out of HBASE-0.90.0 version. If you can move
out from that version then it should work fine I feel.
Regards
Ram
-----Original Message-----
From: V.Sriram [mailto:sriram.v@ask.com]
Sent: Thursday, November 03, 2011 1:11 PM
To: hbase-user@hadoop.apache.org
Subject: RE: RE: Hbase Multiple Master issue
Thats great news. Thanks for ur valuable info!! I will use that =)
Stuti Awasthi wrote:
>
> Hi Sriram,
> I am using Hbase 0.90.3 and I also tested Multiple master issue. It
> worked fine for my cluster. Once new Master is up, after sometime
> region server start communicating with new Master.
> Hope this helps.
>
> -----Original Message-----
> From: V.Sriram [mailto:sriram.v@ask.com]
> Sent: Thursday, November 03, 2011 12:54 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: RE: Hbase Multiple Master issue
>
>
> I saw the release notes of Hbase 0.90.2, 0.90.3 and 0.90.4. This issue
> doesn't seem to be mentioned there. Also since our team want continue
> with 0.90.0 itself for some more time, I am not currently upgrading to
> newer versions.
>
> Today saw the Zk_dump to verify if the new master is currently
> reflected in Zookeeper. It is correctly reflected. But even then the
> region servers don't communicate with the new master.
>
> I think region servers would ideally communicate with zookeeper and
> get the new master node and then communicate with it. But thats not
> happenning!!!
> Any suggestions plz !!
>
>
> V.Sriram wrote:
>>
>> Thanks for the reply. Will try other versions as well!!
>>
>> regards,
>> V.Sriram
>>
>>
>> Ramkrishna S Vasudevan wrote:
>>>
>>> Hi
>>>
>>> If i remember correctly this issue was fixed in later versions of
>>> 0.90.0.
>>> Could you try this in other versions and see if this problem
>>> persists..ideally it should not.
>>>
>>>
>>> Regards
>>> Ram
>>>
>>>
>>>
>>> ----- Original Message -----
>>> From: "V.Sriram" <sr...@ask.com>
>>> Date: Tuesday, November 1, 2011 3:01 pm
>>> Subject: RE: Hbase Multiple Master issue
>>> To: hbase-user@hadoop.apache.org
>>>
>>>>
>>>> I am using 0.90.0 version
>>>>
>>>> regards,
>>>> V.Sriram
>>>>
>>>>
>>>> Ramkrishna S Vasudevan wrote:
>>>> >
>>>> > Hi
>>>> >
>>>> > What version of HBase are you using?
>>>> >
>>>> > Regards
>>>> > Ram
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: V.Sriram [mailto:sriram.v@ask.com]
>>>> > Sent: Tuesday, November 01, 2011 12:58 PM
>>>> > To: hbase-user@hadoop.apache.org
>>>> > Subject: Hbase Multiple Master issue
>>>> >
>>>> >
>>>> > Hi all,
>>>> >
>>>> > I am evaluating HBase Multiple master utility as I require that
>>>> > funtionality. I used a simple 4 machine hbase cluster with M1 as
>>>> master.> Now
>>>> > I started the Hmaster in M4 as well. I could see logs in M4
>>>> Master log as,
>>>> > This HMaster is waiting as already M1 is acting as HMaster.
>>>> >
>>>> > After this I stopped the Master in the M1 and after few seconds,
>>>> the M4
>>>> > master took over. But I am not seeing any region server
>>>> communicating to
>>>> > this new master. The region server logs show,
>>>> >
>>>> > 2011-10-31 23:21:18,834 WARN
>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to
>>>> connect to
>>>> > master. Retrying. Error was:
>>>> > java.net.ConnectException: Connection refused
>>>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>> > at
>>>> >
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:5
>>>> 74)>
>>>> at
>>>> >
>>>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeo
>>>> u
>>>> t.java:2>
>>>> 06)
>>>> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(H
>>>> B
>>>> aseClien>
>>>> t.java:311)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.j
>>>> ava:865)>
>>>> at
>>>> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:2
>>>> 57)> at $Proxy3.getProtocolVersion(Unknown Source)
>>>> > at
>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>>>> > at
>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>>>> > at
>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>>>> > at
>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegio
>>>> n
>>>> Server.j>
>>>> ava:1443)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerR
>>>> e
>>>> port(HRe>
>>>> gionServer.java:737)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServe
>>>> r
>>>> .java:58>
>>>> 6)
>>>> > at java.lang.Thread.run(Thread.java:619)
>>>> >
>>>> > Also in the M4 Master log I am seeing,
>>>> >
>>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>>> > listener.getLocalPort() returned 60010
>>>> > webServer.getConnectors()[0].getLocalPort() returned 60010
>>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>>> Jetty> bound
>>>> > to port 60010
>>>> > 2011-10-31 23:20:47,671 INFO org.mortbay.log: jetty-6.1.26
>>>> > 2011-10-31 23:20:48,691 INFO org.mortbay.log: Started
>>>> > SelectChannelConnector@0.0.0.0:60010
>>>> > 2011-10-31 23:20:48,691 DEBUG
>>>> org.apache.hadoop.hbase.master.HMaster:> Started service threads
>>>> > 2011-10-31 23:20:50,192 INFO
>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>> regionserver(s) to checkin
>>>> > 2011-10-31 23:20:51,695 INFO
>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>> regionserver(s) to checkin
>>>> > 2011-10-31 23:20:53,197 INFO
>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>> regionserver(s) to checkin
>>>> >
>>>> > Seems the region servers are only communicating to the M1 Hmaster
>>>> and not
>>>> > to
>>>> > the new one!!
>>>> >
>>>> > Any help regarding this is highly appreciated :handshake:
>>>> > --
>>>> > View this message in context:
>>>> > http://old.nabble.com/Hbase-Multiple-Master-issue-
>>>> tp32757197p32757197.html> Sent from the HBase User mailing list
>>>> archive at Nabble.com.
>>>> >
>>>> >
>>>> >
>>>>
>>>> --
>>>> View this message in context: http://old.nabble.com/Hbase-Multiple-
>>>> Master-issue-tp32757197p32757528.html
>>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>
>>
>
> --
> View this message in context:
> http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32770898.
> html Sent from the HBase User mailing list archive at Nabble.com.
>
>
> ::DISCLAIMER::
> ----------------------------------------------------------------------
> -------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> It shall not attach any liability on the originator or HCL or its
> affiliates. Any views or opinions presented in this email are solely
> those of the author and may not necessarily reflect the opinions of
> HCL or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure,
> modification, distribution and / or publication of this message
> without the prior written consent of the author of this e-mail is
> strictly prohibited. If you have received this email in error please
> delete it and notify the sender immediately. Before opening any mail
> and attachments please check them for viruses and defect.
>
> ----------------------------------------------------------------------
> -------------------------------------------------
>
>
--
View this message in context:
http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32770950.html
Sent from the HBase User mailing list archive at Nabble.com.
RE: RE: Hbase Multiple Master issue
Posted by "V.Sriram" <sr...@ask.com>.
Thank a lot for your suggesstions Ram. I will see the code. Meanwhile I just
now evaluated with Hbase 0.90.4 and it works fine after throwing few
exceptions - Socket connections with old master then,
2011-11-03 02:14:49,578 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: RemoteException
connecting to master
java.io.IOException: java.io.IOException: java.lang.NullPointerException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:80)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:734)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:595)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
java.lang.NullPointerException
at
org.apache.hadoop.hbase.master.ServerManager.raceThatShouldNotHappenAnymore(ServerManager.java:307)
at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:273)
at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:638)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy5.regionServerReport(Unknown Source)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:728)
... 2 more
After this exception, I got messages like,
Attempting to connect to Master at M4 and then,
Connected to master at M4.
Ramkrishna S Vasudevan wrote:
>
>
>
> If you still cannot move out of 0.90.0 try to understand the code in
> HRegionServer.reportForDuty() in 0.90.3 and merge it in your copy of
> 0.90.0
> and try. Just a suggestion :)
>
> Regards
> Ram
>
>
> -----Original Message-----
> From: V.Sriram [mailto:sriram.v@ask.com]
> Sent: Thursday, November 03, 2011 1:11 PM
> To: hbase-user@hadoop.apache.org
> Subject: RE: RE: Hbase Multiple Master issue
>
>
> Thats great news. Thanks for ur valuable info!! I will use that =)
>
> Stuti Awasthi wrote:
>>
>> Hi Sriram,
>> I am using Hbase 0.90.3 and I also tested Multiple master issue. It
>> worked fine for my cluster. Once new Master is up, after sometime
>> region server start communicating with new Master.
>> Hope this helps.
>>
>> -----Original Message-----
>> From: V.Sriram [mailto:sriram.v@ask.com]
>> Sent: Thursday, November 03, 2011 12:54 PM
>> To: hbase-user@hadoop.apache.org
>> Subject: Re: RE: Hbase Multiple Master issue
>>
>>
>> I saw the release notes of Hbase 0.90.2, 0.90.3 and 0.90.4. This issue
>> doesn't seem to be mentioned there. Also since our team want continue
>> with 0.90.0 itself for some more time, I am not currently upgrading to
>> newer versions.
>>
>> Today saw the Zk_dump to verify if the new master is currently
>> reflected in Zookeeper. It is correctly reflected. But even then the
>> region servers don't communicate with the new master.
>>
>> I think region servers would ideally communicate with zookeeper and
>> get the new master node and then communicate with it. But thats not
>> happenning!!!
>> Any suggestions plz !!
>>
>>
>> V.Sriram wrote:
>>>
>>> Thanks for the reply. Will try other versions as well!!
>>>
>>> regards,
>>> V.Sriram
>>>
>>>
>>> Ramkrishna S Vasudevan wrote:
>>>>
>>>> Hi
>>>>
>>>> If i remember correctly this issue was fixed in later versions of
>>>> 0.90.0.
>>>> Could you try this in other versions and see if this problem
>>>> persists..ideally it should not.
>>>>
>>>>
>>>> Regards
>>>> Ram
>>>>
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: "V.Sriram" <sr...@ask.com>
>>>> Date: Tuesday, November 1, 2011 3:01 pm
>>>> Subject: RE: Hbase Multiple Master issue
>>>> To: hbase-user@hadoop.apache.org
>>>>
>>>>>
>>>>> I am using 0.90.0 version
>>>>>
>>>>> regards,
>>>>> V.Sriram
>>>>>
>>>>>
>>>>> Ramkrishna S Vasudevan wrote:
>>>>> >
>>>>> > Hi
>>>>> >
>>>>> > What version of HBase are you using?
>>>>> >
>>>>> > Regards
>>>>> > Ram
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: V.Sriram [mailto:sriram.v@ask.com]
>>>>> > Sent: Tuesday, November 01, 2011 12:58 PM
>>>>> > To: hbase-user@hadoop.apache.org
>>>>> > Subject: Hbase Multiple Master issue
>>>>> >
>>>>> >
>>>>> > Hi all,
>>>>> >
>>>>> > I am evaluating HBase Multiple master utility as I require that
>>>>> > funtionality. I used a simple 4 machine hbase cluster with M1 as
>>>>> master.> Now
>>>>> > I started the Hmaster in M4 as well. I could see logs in M4
>>>>> Master log as,
>>>>> > This HMaster is waiting as already M1 is acting as HMaster.
>>>>> >
>>>>> > After this I stopped the Master in the M1 and after few seconds,
>>>>> the M4
>>>>> > master took over. But I am not seeing any region server
>>>>> communicating to
>>>>> > this new master. The region server logs show,
>>>>> >
>>>>> > 2011-10-31 23:21:18,834 WARN
>>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to
>>>>> connect to
>>>>> > master. Retrying. Error was:
>>>>> > java.net.ConnectException: Connection refused
>>>>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>> > at
>>>>> >
>>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:5
>>>>> 74)>
>>>>> at
>>>>> >
>>>>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeo
>>>>> u
>>>>> t.java:2>
>>>>> 06)
>>>>> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(H
>>>>> B
>>>>> aseClien>
>>>>> t.java:311)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.j
>>>>> ava:865)>
>>>>> at
>>>>> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:2
>>>>> 57)> at $Proxy3.getProtocolVersion(Unknown Source)
>>>>> > at
>>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>>>>> > at
>>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>>>>> > at
>>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>>>>> > at
>>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegio
>>>>> n
>>>>> Server.j>
>>>>> ava:1443)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerR
>>>>> e
>>>>> port(HRe>
>>>>> gionServer.java:737)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServe
>>>>> r
>>>>> .java:58>
>>>>> 6)
>>>>> > at java.lang.Thread.run(Thread.java:619)
>>>>> >
>>>>> > Also in the M4 Master log I am seeing,
>>>>> >
>>>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>>>> > listener.getLocalPort() returned 60010
>>>>> > webServer.getConnectors()[0].getLocalPort() returned 60010
>>>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>>>> Jetty> bound
>>>>> > to port 60010
>>>>> > 2011-10-31 23:20:47,671 INFO org.mortbay.log: jetty-6.1.26
>>>>> > 2011-10-31 23:20:48,691 INFO org.mortbay.log: Started
>>>>> > SelectChannelConnector@0.0.0.0:60010
>>>>> > 2011-10-31 23:20:48,691 DEBUG
>>>>> org.apache.hadoop.hbase.master.HMaster:> Started service threads
>>>>> > 2011-10-31 23:20:50,192 INFO
>>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>>> regionserver(s) to checkin
>>>>> > 2011-10-31 23:20:51,695 INFO
>>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>>> regionserver(s) to checkin
>>>>> > 2011-10-31 23:20:53,197 INFO
>>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>>> regionserver(s) to checkin
>>>>> >
>>>>> > Seems the region servers are only communicating to the M1 Hmaster
>>>>> and not
>>>>> > to
>>>>> > the new one!!
>>>>> >
>>>>> > Any help regarding this is highly appreciated :handshake:
>>>>> > --
>>>>> > View this message in context:
>>>>> > http://old.nabble.com/Hbase-Multiple-Master-issue-
>>>>> tp32757197p32757197.html> Sent from the HBase User mailing list
>>>>> archive at Nabble.com.
>>>>> >
>>>>> >
>>>>> >
>>>>>
>>>>> --
>>>>> View this message in context: http://old.nabble.com/Hbase-Multiple-
>>>>> Master-issue-tp32757197p32757528.html
>>>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32770898.
>> html Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>> ::DISCLAIMER::
>> ----------------------------------------------------------------------
>> -------------------------------------------------
>>
>> The contents of this e-mail and any attachment(s) are confidential and
>> intended for the named recipient(s) only.
>> It shall not attach any liability on the originator or HCL or its
>> affiliates. Any views or opinions presented in this email are solely
>> those of the author and may not necessarily reflect the opinions of
>> HCL or its affiliates.
>> Any form of reproduction, dissemination, copying, disclosure,
>> modification, distribution and / or publication of this message
>> without the prior written consent of the author of this e-mail is
>> strictly prohibited. If you have received this email in error please
>> delete it and notify the sender immediately. Before opening any mail
>> and attachments please check them for viruses and defect.
>>
>> ----------------------------------------------------------------------
>> -------------------------------------------------
>>
>>
>
> --
> View this message in context:
> http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32770950.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32771444.html
Sent from the HBase User mailing list archive at Nabble.com.
RE: RE: Hbase Multiple Master issue
Posted by Ramkrishna S Vasudevan <ra...@huawei.com>.
If you still cannot move out of 0.90.0 try to understand the code in
HRegionServer.reportForDuty() in 0.90.3 and merge it in your copy of 0.90.0
and try. Just a suggestion :)
Regards
Ram
-----Original Message-----
From: V.Sriram [mailto:sriram.v@ask.com]
Sent: Thursday, November 03, 2011 1:11 PM
To: hbase-user@hadoop.apache.org
Subject: RE: RE: Hbase Multiple Master issue
Thats great news. Thanks for ur valuable info!! I will use that =)
Stuti Awasthi wrote:
>
> Hi Sriram,
> I am using Hbase 0.90.3 and I also tested Multiple master issue. It
> worked fine for my cluster. Once new Master is up, after sometime
> region server start communicating with new Master.
> Hope this helps.
>
> -----Original Message-----
> From: V.Sriram [mailto:sriram.v@ask.com]
> Sent: Thursday, November 03, 2011 12:54 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: RE: Hbase Multiple Master issue
>
>
> I saw the release notes of Hbase 0.90.2, 0.90.3 and 0.90.4. This issue
> doesn't seem to be mentioned there. Also since our team want continue
> with 0.90.0 itself for some more time, I am not currently upgrading to
> newer versions.
>
> Today saw the Zk_dump to verify if the new master is currently
> reflected in Zookeeper. It is correctly reflected. But even then the
> region servers don't communicate with the new master.
>
> I think region servers would ideally communicate with zookeeper and
> get the new master node and then communicate with it. But thats not
> happenning!!!
> Any suggestions plz !!
>
>
> V.Sriram wrote:
>>
>> Thanks for the reply. Will try other versions as well!!
>>
>> regards,
>> V.Sriram
>>
>>
>> Ramkrishna S Vasudevan wrote:
>>>
>>> Hi
>>>
>>> If i remember correctly this issue was fixed in later versions of
>>> 0.90.0.
>>> Could you try this in other versions and see if this problem
>>> persists..ideally it should not.
>>>
>>>
>>> Regards
>>> Ram
>>>
>>>
>>>
>>> ----- Original Message -----
>>> From: "V.Sriram" <sr...@ask.com>
>>> Date: Tuesday, November 1, 2011 3:01 pm
>>> Subject: RE: Hbase Multiple Master issue
>>> To: hbase-user@hadoop.apache.org
>>>
>>>>
>>>> I am using 0.90.0 version
>>>>
>>>> regards,
>>>> V.Sriram
>>>>
>>>>
>>>> Ramkrishna S Vasudevan wrote:
>>>> >
>>>> > Hi
>>>> >
>>>> > What version of HBase are you using?
>>>> >
>>>> > Regards
>>>> > Ram
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: V.Sriram [mailto:sriram.v@ask.com]
>>>> > Sent: Tuesday, November 01, 2011 12:58 PM
>>>> > To: hbase-user@hadoop.apache.org
>>>> > Subject: Hbase Multiple Master issue
>>>> >
>>>> >
>>>> > Hi all,
>>>> >
>>>> > I am evaluating HBase Multiple master utility as I require that
>>>> > funtionality. I used a simple 4 machine hbase cluster with M1 as
>>>> master.> Now
>>>> > I started the Hmaster in M4 as well. I could see logs in M4
>>>> Master log as,
>>>> > This HMaster is waiting as already M1 is acting as HMaster.
>>>> >
>>>> > After this I stopped the Master in the M1 and after few seconds,
>>>> the M4
>>>> > master took over. But I am not seeing any region server
>>>> communicating to
>>>> > this new master. The region server logs show,
>>>> >
>>>> > 2011-10-31 23:21:18,834 WARN
>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to
>>>> connect to
>>>> > master. Retrying. Error was:
>>>> > java.net.ConnectException: Connection refused
>>>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>> > at
>>>> >
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:5
>>>> 74)>
>>>> at
>>>> >
>>>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeo
>>>> u
>>>> t.java:2>
>>>> 06)
>>>> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(H
>>>> B
>>>> aseClien>
>>>> t.java:311)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.j
>>>> ava:865)>
>>>> at
>>>> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:2
>>>> 57)> at $Proxy3.getProtocolVersion(Unknown Source)
>>>> > at
>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>>>> > at
>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>>>> > at
>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>>>> > at
>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegio
>>>> n
>>>> Server.j>
>>>> ava:1443)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerR
>>>> e
>>>> port(HRe>
>>>> gionServer.java:737)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServe
>>>> r
>>>> .java:58>
>>>> 6)
>>>> > at java.lang.Thread.run(Thread.java:619)
>>>> >
>>>> > Also in the M4 Master log I am seeing,
>>>> >
>>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>>> > listener.getLocalPort() returned 60010
>>>> > webServer.getConnectors()[0].getLocalPort() returned 60010
>>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>>> Jetty> bound
>>>> > to port 60010
>>>> > 2011-10-31 23:20:47,671 INFO org.mortbay.log: jetty-6.1.26
>>>> > 2011-10-31 23:20:48,691 INFO org.mortbay.log: Started
>>>> > SelectChannelConnector@0.0.0.0:60010
>>>> > 2011-10-31 23:20:48,691 DEBUG
>>>> org.apache.hadoop.hbase.master.HMaster:> Started service threads
>>>> > 2011-10-31 23:20:50,192 INFO
>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>> regionserver(s) to checkin
>>>> > 2011-10-31 23:20:51,695 INFO
>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>> regionserver(s) to checkin
>>>> > 2011-10-31 23:20:53,197 INFO
>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>> regionserver(s) to checkin
>>>> >
>>>> > Seems the region servers are only communicating to the M1 Hmaster
>>>> and not
>>>> > to
>>>> > the new one!!
>>>> >
>>>> > Any help regarding this is highly appreciated :handshake:
>>>> > --
>>>> > View this message in context:
>>>> > http://old.nabble.com/Hbase-Multiple-Master-issue-
>>>> tp32757197p32757197.html> Sent from the HBase User mailing list
>>>> archive at Nabble.com.
>>>> >
>>>> >
>>>> >
>>>>
>>>> --
>>>> View this message in context: http://old.nabble.com/Hbase-Multiple-
>>>> Master-issue-tp32757197p32757528.html
>>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>
>>
>
> --
> View this message in context:
> http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32770898.
> html Sent from the HBase User mailing list archive at Nabble.com.
>
>
> ::DISCLAIMER::
> ----------------------------------------------------------------------
> -------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> It shall not attach any liability on the originator or HCL or its
> affiliates. Any views or opinions presented in this email are solely
> those of the author and may not necessarily reflect the opinions of
> HCL or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure,
> modification, distribution and / or publication of this message
> without the prior written consent of the author of this e-mail is
> strictly prohibited. If you have received this email in error please
> delete it and notify the sender immediately. Before opening any mail
> and attachments please check them for viruses and defect.
>
> ----------------------------------------------------------------------
> -------------------------------------------------
>
>
--
View this message in context:
http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32770950.html
Sent from the HBase User mailing list archive at Nabble.com.
RE: RE: Hbase Multiple Master issue
Posted by "V.Sriram" <sr...@ask.com>.
Thats great news. Thanks for ur valuable info!! I will use that =)
Stuti Awasthi wrote:
>
> Hi Sriram,
> I am using Hbase 0.90.3 and I also tested Multiple master issue. It
> worked fine for my cluster. Once new Master is up, after sometime region
> server start communicating with new Master.
> Hope this helps.
>
> -----Original Message-----
> From: V.Sriram [mailto:sriram.v@ask.com]
> Sent: Thursday, November 03, 2011 12:54 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: RE: Hbase Multiple Master issue
>
>
> I saw the release notes of Hbase 0.90.2, 0.90.3 and 0.90.4. This issue
> doesn't seem to be mentioned there. Also since our team want continue with
> 0.90.0 itself for some more time, I am not currently upgrading to newer
> versions.
>
> Today saw the Zk_dump to verify if the new master is currently reflected
> in Zookeeper. It is correctly reflected. But even then the region servers
> don't communicate with the new master.
>
> I think region servers would ideally communicate with zookeeper and get
> the new master node and then communicate with it. But thats not
> happenning!!!
> Any suggestions plz !!
>
>
> V.Sriram wrote:
>>
>> Thanks for the reply. Will try other versions as well!!
>>
>> regards,
>> V.Sriram
>>
>>
>> Ramkrishna S Vasudevan wrote:
>>>
>>> Hi
>>>
>>> If i remember correctly this issue was fixed in later versions of
>>> 0.90.0.
>>> Could you try this in other versions and see if this problem
>>> persists..ideally it should not.
>>>
>>>
>>> Regards
>>> Ram
>>>
>>>
>>>
>>> ----- Original Message -----
>>> From: "V.Sriram" <sr...@ask.com>
>>> Date: Tuesday, November 1, 2011 3:01 pm
>>> Subject: RE: Hbase Multiple Master issue
>>> To: hbase-user@hadoop.apache.org
>>>
>>>>
>>>> I am using 0.90.0 version
>>>>
>>>> regards,
>>>> V.Sriram
>>>>
>>>>
>>>> Ramkrishna S Vasudevan wrote:
>>>> >
>>>> > Hi
>>>> >
>>>> > What version of HBase are you using?
>>>> >
>>>> > Regards
>>>> > Ram
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: V.Sriram [mailto:sriram.v@ask.com]
>>>> > Sent: Tuesday, November 01, 2011 12:58 PM
>>>> > To: hbase-user@hadoop.apache.org
>>>> > Subject: Hbase Multiple Master issue
>>>> >
>>>> >
>>>> > Hi all,
>>>> >
>>>> > I am evaluating HBase Multiple master utility as I require that
>>>> > funtionality. I used a simple 4 machine hbase cluster with M1 as
>>>> master.> Now
>>>> > I started the Hmaster in M4 as well. I could see logs in M4
>>>> Master log as,
>>>> > This HMaster is waiting as already M1 is acting as HMaster.
>>>> >
>>>> > After this I stopped the Master in the M1 and after few seconds,
>>>> the M4
>>>> > master took over. But I am not seeing any region server
>>>> communicating to
>>>> > this new master. The region server logs show,
>>>> >
>>>> > 2011-10-31 23:21:18,834 WARN
>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to
>>>> connect to
>>>> > master. Retrying. Error was:
>>>> > java.net.ConnectException: Connection refused
>>>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>> > at
>>>> >
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)>
>>>> at
>>>> >
>>>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeou
>>>> t.java:2>
>>>> 06)
>>>> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HB
>>>> aseClien>
>>>> t.java:311)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)>
>>>> at
>>>> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)>
>>>> at $Proxy3.getProtocolVersion(Unknown Source)
>>>> > at
>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>>>> > at
>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>>>> > at
>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>>>> > at
>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegion
>>>> Server.j>
>>>> ava:1443)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerRe
>>>> port(HRe>
>>>> gionServer.java:737)
>>>> > at
>>>> >
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer
>>>> .java:58>
>>>> 6)
>>>> > at java.lang.Thread.run(Thread.java:619)
>>>> >
>>>> > Also in the M4 Master log I am seeing,
>>>> >
>>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>>> > listener.getLocalPort() returned 60010
>>>> > webServer.getConnectors()[0].getLocalPort() returned 60010
>>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>>> Jetty> bound
>>>> > to port 60010
>>>> > 2011-10-31 23:20:47,671 INFO org.mortbay.log: jetty-6.1.26
>>>> > 2011-10-31 23:20:48,691 INFO org.mortbay.log: Started
>>>> > SelectChannelConnector@0.0.0.0:60010
>>>> > 2011-10-31 23:20:48,691 DEBUG
>>>> org.apache.hadoop.hbase.master.HMaster:> Started service threads
>>>> > 2011-10-31 23:20:50,192 INFO
>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>> regionserver(s) to checkin
>>>> > 2011-10-31 23:20:51,695 INFO
>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>> regionserver(s) to checkin
>>>> > 2011-10-31 23:20:53,197 INFO
>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>> regionserver(s) to checkin
>>>> >
>>>> > Seems the region servers are only communicating to the M1 Hmaster
>>>> and not
>>>> > to
>>>> > the new one!!
>>>> >
>>>> > Any help regarding this is highly appreciated :handshake:
>>>> > --
>>>> > View this message in context:
>>>> > http://old.nabble.com/Hbase-Multiple-Master-issue-
>>>> tp32757197p32757197.html> Sent from the HBase User mailing list
>>>> archive at Nabble.com.
>>>> >
>>>> >
>>>> >
>>>>
>>>> --
>>>> View this message in context: http://old.nabble.com/Hbase-Multiple-
>>>> Master-issue-tp32757197p32757528.html
>>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>
>>
>
> --
> View this message in context:
> http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32770898.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
> ::DISCLAIMER::
> -----------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> It shall not attach any liability on the originator or HCL or its
> affiliates. Any views or opinions presented in
> this email are solely those of the author and may not necessarily reflect
> the opinions of HCL or its affiliates.
> Any form of reproduction, dissemination, copying, disclosure,
> modification, distribution and / or publication of
> this message without the prior written consent of the author of this
> e-mail is strictly prohibited. If you have
> received this email in error please delete it and notify the sender
> immediately. Before opening any mail and
> attachments please check them for viruses and defect.
>
> -----------------------------------------------------------------------------------------------------------------------
>
>
--
View this message in context: http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32770950.html
Sent from the HBase User mailing list archive at Nabble.com.
RE: RE: Hbase Multiple Master issue
Posted by Stuti Awasthi <st...@hcl.com>.
Hi Sriram,
I am using Hbase 0.90.3 and I also tested Multiple master issue. It worked fine for my cluster. Once new Master is up, after sometime region server start communicating with new Master.
Hope this helps.
-----Original Message-----
From: V.Sriram [mailto:sriram.v@ask.com]
Sent: Thursday, November 03, 2011 12:54 PM
To: hbase-user@hadoop.apache.org
Subject: Re: RE: Hbase Multiple Master issue
I saw the release notes of Hbase 0.90.2, 0.90.3 and 0.90.4. This issue doesn't seem to be mentioned there. Also since our team want continue with
0.90.0 itself for some more time, I am not currently upgrading to newer versions.
Today saw the Zk_dump to verify if the new master is currently reflected in Zookeeper. It is correctly reflected. But even then the region servers don't communicate with the new master.
I think region servers would ideally communicate with zookeeper and get the new master node and then communicate with it. But thats not happenning!!!
Any suggestions plz !!
V.Sriram wrote:
>
> Thanks for the reply. Will try other versions as well!!
>
> regards,
> V.Sriram
>
>
> Ramkrishna S Vasudevan wrote:
>>
>> Hi
>>
>> If i remember correctly this issue was fixed in later versions of
>> 0.90.0.
>> Could you try this in other versions and see if this problem
>> persists..ideally it should not.
>>
>>
>> Regards
>> Ram
>>
>>
>>
>> ----- Original Message -----
>> From: "V.Sriram" <sr...@ask.com>
>> Date: Tuesday, November 1, 2011 3:01 pm
>> Subject: RE: Hbase Multiple Master issue
>> To: hbase-user@hadoop.apache.org
>>
>>>
>>> I am using 0.90.0 version
>>>
>>> regards,
>>> V.Sriram
>>>
>>>
>>> Ramkrishna S Vasudevan wrote:
>>> >
>>> > Hi
>>> >
>>> > What version of HBase are you using?
>>> >
>>> > Regards
>>> > Ram
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: V.Sriram [mailto:sriram.v@ask.com]
>>> > Sent: Tuesday, November 01, 2011 12:58 PM
>>> > To: hbase-user@hadoop.apache.org
>>> > Subject: Hbase Multiple Master issue
>>> >
>>> >
>>> > Hi all,
>>> >
>>> > I am evaluating HBase Multiple master utility as I require that
>>> > funtionality. I used a simple 4 machine hbase cluster with M1 as
>>> master.> Now
>>> > I started the Hmaster in M4 as well. I could see logs in M4
>>> Master log as,
>>> > This HMaster is waiting as already M1 is acting as HMaster.
>>> >
>>> > After this I stopped the Master in the M1 and after few seconds,
>>> the M4
>>> > master took over. But I am not seeing any region server
>>> communicating to
>>> > this new master. The region server logs show,
>>> >
>>> > 2011-10-31 23:21:18,834 WARN
>>> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to
>>> connect to
>>> > master. Retrying. Error was:
>>> > java.net.ConnectException: Connection refused
>>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>> > at
>>> >
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)>
>>> at
>>> >
>>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeou
>>> t.java:2>
>>> 06)
>>> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HB
>>> aseClien>
>>> t.java:311)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)>
>>> at
>>> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)>
>>> at $Proxy3.getProtocolVersion(Unknown Source)
>>> > at
>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>>> > at
>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>>> > at
>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>>> > at
>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegion
>>> Server.j>
>>> ava:1443)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerRe
>>> port(HRe>
>>> gionServer.java:737)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer
>>> .java:58>
>>> 6)
>>> > at java.lang.Thread.run(Thread.java:619)
>>> >
>>> > Also in the M4 Master log I am seeing,
>>> >
>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>> > listener.getLocalPort() returned 60010
>>> > webServer.getConnectors()[0].getLocalPort() returned 60010
>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>> Jetty> bound
>>> > to port 60010
>>> > 2011-10-31 23:20:47,671 INFO org.mortbay.log: jetty-6.1.26
>>> > 2011-10-31 23:20:48,691 INFO org.mortbay.log: Started
>>> > SelectChannelConnector@0.0.0.0:60010
>>> > 2011-10-31 23:20:48,691 DEBUG
>>> org.apache.hadoop.hbase.master.HMaster:> Started service threads
>>> > 2011-10-31 23:20:50,192 INFO
>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>> regionserver(s) to checkin
>>> > 2011-10-31 23:20:51,695 INFO
>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>> regionserver(s) to checkin
>>> > 2011-10-31 23:20:53,197 INFO
>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>> regionserver(s) to checkin
>>> >
>>> > Seems the region servers are only communicating to the M1 Hmaster
>>> and not
>>> > to
>>> > the new one!!
>>> >
>>> > Any help regarding this is highly appreciated :handshake:
>>> > --
>>> > View this message in context:
>>> > http://old.nabble.com/Hbase-Multiple-Master-issue-
>>> tp32757197p32757197.html> Sent from the HBase User mailing list
>>> archive at Nabble.com.
>>> >
>>> >
>>> >
>>>
>>> --
>>> View this message in context: http://old.nabble.com/Hbase-Multiple-
>>> Master-issue-tp32757197p32757528.html
>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>
>
--
View this message in context: http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32770898.html
Sent from the HBase User mailing list archive at Nabble.com.
::DISCLAIMER::
-----------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in
this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of
this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have
received this email in error please delete it and notify the sender immediately. Before opening any mail and
attachments please check them for viruses and defect.
-----------------------------------------------------------------------------------------------------------------------
Re: RE: Hbase Multiple Master issue
Posted by "V.Sriram" <sr...@ask.com>.
I saw the release notes of Hbase 0.90.2, 0.90.3 and 0.90.4. This issue
doesn't seem to be mentioned there. Also since our team want continue with
0.90.0 itself for some more time, I am not currently upgrading to newer
versions.
Today saw the Zk_dump to verify if the new master is currently reflected in
Zookeeper. It is correctly reflected. But even then the region servers don't
communicate with the new master.
I think region servers would ideally communicate with zookeeper and get the
new master node and then communicate with it. But thats not happenning!!!
Any suggestions plz !!
V.Sriram wrote:
>
> Thanks for the reply. Will try other versions as well!!
>
> regards,
> V.Sriram
>
>
> Ramkrishna S Vasudevan wrote:
>>
>> Hi
>>
>> If i remember correctly this issue was fixed in later versions of
>> 0.90.0.
>> Could you try this in other versions and see if this problem
>> persists..ideally it should not.
>>
>>
>> Regards
>> Ram
>>
>>
>>
>> ----- Original Message -----
>> From: "V.Sriram" <sr...@ask.com>
>> Date: Tuesday, November 1, 2011 3:01 pm
>> Subject: RE: Hbase Multiple Master issue
>> To: hbase-user@hadoop.apache.org
>>
>>>
>>> I am using 0.90.0 version
>>>
>>> regards,
>>> V.Sriram
>>>
>>>
>>> Ramkrishna S Vasudevan wrote:
>>> >
>>> > Hi
>>> >
>>> > What version of HBase are you using?
>>> >
>>> > Regards
>>> > Ram
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: V.Sriram [mailto:sriram.v@ask.com]
>>> > Sent: Tuesday, November 01, 2011 12:58 PM
>>> > To: hbase-user@hadoop.apache.org
>>> > Subject: Hbase Multiple Master issue
>>> >
>>> >
>>> > Hi all,
>>> >
>>> > I am evaluating HBase Multiple master utility as I require that
>>> > funtionality. I used a simple 4 machine hbase cluster with M1 as
>>> master.> Now
>>> > I started the Hmaster in M4 as well. I could see logs in M4
>>> Master log as,
>>> > This HMaster is waiting as already M1 is acting as HMaster.
>>> >
>>> > After this I stopped the Master in the M1 and after few seconds,
>>> the M4
>>> > master took over. But I am not seeing any region server
>>> communicating to
>>> > this new master. The region server logs show,
>>> >
>>> > 2011-10-31 23:21:18,834 WARN
>>> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to
>>> connect to
>>> > master. Retrying. Error was:
>>> > java.net.ConnectException: Connection refused
>>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>> > at
>>> >
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)>
>>> at
>>> >
>>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:2>
>>> 06)
>>> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClien>
>>> t.java:311)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)>
>>> at
>>> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)>
>>> at $Proxy3.getProtocolVersion(Unknown Source)
>>> > at
>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>>> > at
>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>>> > at
>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>>> > at
>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.j>
>>> ava:1443)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRe>
>>> gionServer.java:737)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:58>
>>> 6)
>>> > at java.lang.Thread.run(Thread.java:619)
>>> >
>>> > Also in the M4 Master log I am seeing,
>>> >
>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>> > listener.getLocalPort() returned 60010
>>> > webServer.getConnectors()[0].getLocalPort() returned 60010
>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>> Jetty> bound
>>> > to port 60010
>>> > 2011-10-31 23:20:47,671 INFO org.mortbay.log: jetty-6.1.26
>>> > 2011-10-31 23:20:48,691 INFO org.mortbay.log: Started
>>> > SelectChannelConnector@0.0.0.0:60010
>>> > 2011-10-31 23:20:48,691 DEBUG
>>> org.apache.hadoop.hbase.master.HMaster:> Started service threads
>>> > 2011-10-31 23:20:50,192 INFO
>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>> regionserver(s) to checkin
>>> > 2011-10-31 23:20:51,695 INFO
>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>> regionserver(s) to checkin
>>> > 2011-10-31 23:20:53,197 INFO
>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>> regionserver(s) to checkin
>>> >
>>> > Seems the region servers are only communicating to the M1 Hmaster
>>> and not
>>> > to
>>> > the new one!!
>>> >
>>> > Any help regarding this is highly appreciated :handshake:
>>> > --
>>> > View this message in context:
>>> > http://old.nabble.com/Hbase-Multiple-Master-issue-
>>> tp32757197p32757197.html> Sent from the HBase User mailing list
>>> archive at Nabble.com.
>>> >
>>> >
>>> >
>>>
>>> --
>>> View this message in context: http://old.nabble.com/Hbase-Multiple-
>>> Master-issue-tp32757197p32757528.html
>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>
>
--
View this message in context: http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32770898.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: RE: Hbase Multiple Master issue
Posted by "V.Sriram" <sr...@ask.com>.
Thanks for the reply. Will try other versions as well!!
regards,
V.Sriram
Ramkrishna S Vasudevan wrote:
>
> Hi
>
> If i remember correctly this issue was fixed in later versions of 0.90.0.
> Could you try this in other versions and see if this problem
> persists..ideally it should not.
>
>
> Regards
> Ram
>
>
>
> ----- Original Message -----
> From: "V.Sriram" <sr...@ask.com>
> Date: Tuesday, November 1, 2011 3:01 pm
> Subject: RE: Hbase Multiple Master issue
> To: hbase-user@hadoop.apache.org
>
>>
>> I am using 0.90.0 version
>>
>> regards,
>> V.Sriram
>>
>>
>> Ramkrishna S Vasudevan wrote:
>> >
>> > Hi
>> >
>> > What version of HBase are you using?
>> >
>> > Regards
>> > Ram
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: V.Sriram [mailto:sriram.v@ask.com]
>> > Sent: Tuesday, November 01, 2011 12:58 PM
>> > To: hbase-user@hadoop.apache.org
>> > Subject: Hbase Multiple Master issue
>> >
>> >
>> > Hi all,
>> >
>> > I am evaluating HBase Multiple master utility as I require that
>> > funtionality. I used a simple 4 machine hbase cluster with M1 as
>> master.> Now
>> > I started the Hmaster in M4 as well. I could see logs in M4
>> Master log as,
>> > This HMaster is waiting as already M1 is acting as HMaster.
>> >
>> > After this I stopped the Master in the M1 and after few seconds,
>> the M4
>> > master took over. But I am not seeing any region server
>> communicating to
>> > this new master. The region server logs show,
>> >
>> > 2011-10-31 23:21:18,834 WARN
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to
>> connect to
>> > master. Retrying. Error was:
>> > java.net.ConnectException: Connection refused
>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> > at
>> >
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)>
>> at
>> >
>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:2>
>> 06)
>> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClien>
>> t.java:311)
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)>
>> at
>> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)>
>> at $Proxy3.getProtocolVersion(Unknown Source)
>> > at
>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>> > at
>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>> > at
>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>> > at
>> > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.j>
>> ava:1443)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRe>
>> gionServer.java:737)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:58>
>> 6)
>> > at java.lang.Thread.run(Thread.java:619)
>> >
>> > Also in the M4 Master log I am seeing,
>> >
>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>> > listener.getLocalPort() returned 60010
>> > webServer.getConnectors()[0].getLocalPort() returned 60010
>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>> Jetty> bound
>> > to port 60010
>> > 2011-10-31 23:20:47,671 INFO org.mortbay.log: jetty-6.1.26
>> > 2011-10-31 23:20:48,691 INFO org.mortbay.log: Started
>> > SelectChannelConnector@0.0.0.0:60010
>> > 2011-10-31 23:20:48,691 DEBUG
>> org.apache.hadoop.hbase.master.HMaster:> Started service threads
>> > 2011-10-31 23:20:50,192 INFO
>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>> regionserver(s) to checkin
>> > 2011-10-31 23:20:51,695 INFO
>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>> regionserver(s) to checkin
>> > 2011-10-31 23:20:53,197 INFO
>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>> regionserver(s) to checkin
>> >
>> > Seems the region servers are only communicating to the M1 Hmaster
>> and not
>> > to
>> > the new one!!
>> >
>> > Any help regarding this is highly appreciated :handshake:
>> > --
>> > View this message in context:
>> > http://old.nabble.com/Hbase-Multiple-Master-issue-
>> tp32757197p32757197.html> Sent from the HBase User mailing list
>> archive at Nabble.com.
>> >
>> >
>> >
>>
>> --
>> View this message in context: http://old.nabble.com/Hbase-Multiple-
>> Master-issue-tp32757197p32757528.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32758542.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: RE: Hbase Multiple Master issue
Posted by Ramakrishna S Vasudevan 00902313 <ra...@huawei.com>.
Hi
If i remember correctly this issue was fixed in later versions of 0.90.0.
Could you try this in other versions and see if this problem persists..ideally it should not.
Regards
Ram
----- Original Message -----
From: "V.Sriram" <sr...@ask.com>
Date: Tuesday, November 1, 2011 3:01 pm
Subject: RE: Hbase Multiple Master issue
To: hbase-user@hadoop.apache.org
>
> I am using 0.90.0 version
>
> regards,
> V.Sriram
>
>
> Ramkrishna S Vasudevan wrote:
> >
> > Hi
> >
> > What version of HBase are you using?
> >
> > Regards
> > Ram
> >
> >
> >
> > -----Original Message-----
> > From: V.Sriram [mailto:sriram.v@ask.com]
> > Sent: Tuesday, November 01, 2011 12:58 PM
> > To: hbase-user@hadoop.apache.org
> > Subject: Hbase Multiple Master issue
> >
> >
> > Hi all,
> >
> > I am evaluating HBase Multiple master utility as I require that
> > funtionality. I used a simple 4 machine hbase cluster with M1 as
> master.> Now
> > I started the Hmaster in M4 as well. I could see logs in M4
> Master log as,
> > This HMaster is waiting as already M1 is acting as HMaster.
> >
> > After this I stopped the Master in the M1 and after few seconds,
> the M4
> > master took over. But I am not seeing any region server
> communicating to
> > this new master. The region server logs show,
> >
> > 2011-10-31 23:21:18,834 WARN
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to
> connect to
> > master. Retrying. Error was:
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> >
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)> at
> >
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:2> 06)
> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClien> t.java:311)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)> at
> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)> at $Proxy3.getProtocolVersion(Unknown Source)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.j> ava:1443)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRe> gionServer.java:737)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:58> 6)
> > at java.lang.Thread.run(Thread.java:619)
> >
> > Also in the M4 Master log I am seeing,
> >
> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
> > listener.getLocalPort() returned 60010
> > webServer.getConnectors()[0].getLocalPort() returned 60010
> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
> Jetty> bound
> > to port 60010
> > 2011-10-31 23:20:47,671 INFO org.mortbay.log: jetty-6.1.26
> > 2011-10-31 23:20:48,691 INFO org.mortbay.log: Started
> > SelectChannelConnector@0.0.0.0:60010
> > 2011-10-31 23:20:48,691 DEBUG
> org.apache.hadoop.hbase.master.HMaster:> Started service threads
> > 2011-10-31 23:20:50,192 INFO
> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
> regionserver(s) to checkin
> > 2011-10-31 23:20:51,695 INFO
> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
> regionserver(s) to checkin
> > 2011-10-31 23:20:53,197 INFO
> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
> regionserver(s) to checkin
> >
> > Seems the region servers are only communicating to the M1 Hmaster
> and not
> > to
> > the new one!!
> >
> > Any help regarding this is highly appreciated :handshake:
> > --
> > View this message in context:
> > http://old.nabble.com/Hbase-Multiple-Master-issue-
> tp32757197p32757197.html> Sent from the HBase User mailing list
> archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context: http://old.nabble.com/Hbase-Multiple-
> Master-issue-tp32757197p32757528.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
RE: Hbase Multiple Master issue
Posted by "V.Sriram" <sr...@ask.com>.
I am using 0.90.0 version
regards,
V.Sriram
Ramkrishna S Vasudevan wrote:
>
> Hi
>
> What version of HBase are you using?
>
> Regards
> Ram
>
>
>
> -----Original Message-----
> From: V.Sriram [mailto:sriram.v@ask.com]
> Sent: Tuesday, November 01, 2011 12:58 PM
> To: hbase-user@hadoop.apache.org
> Subject: Hbase Multiple Master issue
>
>
> Hi all,
>
> I am evaluating HBase Multiple master utility as I require that
> funtionality. I used a simple 4 machine hbase cluster with M1 as master.
> Now
> I started the Hmaster in M4 as well. I could see logs in M4 Master log as,
> This HMaster is waiting as already M1 is acting as HMaster.
>
> After this I stopped the Master in the M1 and after few seconds, the M4
> master took over. But I am not seeing any region server communicating to
> this new master. The region server logs show,
>
> 2011-10-31 23:21:18,834 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
> master. Retrying. Error was:
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:2
> 06)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClien
> t.java:311)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy3.getProtocolVersion(Unknown Source)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.j
> ava:1443)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRe
> gionServer.java:737)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:58
> 6)
> at java.lang.Thread.run(Thread.java:619)
>
> Also in the M4 Master log I am seeing,
>
> 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 60010
> webServer.getConnectors()[0].getLocalPort() returned 60010
> 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound
> to port 60010
> 2011-10-31 23:20:47,671 INFO org.mortbay.log: jetty-6.1.26
> 2011-10-31 23:20:48,691 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:60010
> 2011-10-31 23:20:48,691 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Started service threads
> 2011-10-31 23:20:50,192 INFO org.apache.hadoop.hbase.master.ServerManager:
> Waiting on regionserver(s) to checkin
> 2011-10-31 23:20:51,695 INFO org.apache.hadoop.hbase.master.ServerManager:
> Waiting on regionserver(s) to checkin
> 2011-10-31 23:20:53,197 INFO org.apache.hadoop.hbase.master.ServerManager:
> Waiting on regionserver(s) to checkin
>
> Seems the region servers are only communicating to the M1 Hmaster and not
> to
> the new one!!
>
> Any help regarding this is highly appreciated :handshake:
> --
> View this message in context:
> http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32757197.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32757528.html
Sent from the HBase User mailing list archive at Nabble.com.
RE: Hbase Multiple Master issue
Posted by Ramkrishna S Vasudevan <ra...@huawei.com>.
Hi
What version of HBase are you using?
Regards
Ram
-----Original Message-----
From: V.Sriram [mailto:sriram.v@ask.com]
Sent: Tuesday, November 01, 2011 12:58 PM
To: hbase-user@hadoop.apache.org
Subject: Hbase Multiple Master issue
Hi all,
I am evaluating HBase Multiple master utility as I require that
funtionality. I used a simple 4 machine hbase cluster with M1 as master. Now
I started the Hmaster in M4 as well. I could see logs in M4 Master log as,
This HMaster is waiting as already M1 is acting as HMaster.
After this I stopped the Master in the M1 and after few seconds, the M4
master took over. But I am not seeing any region server communicating to
this new master. The region server logs show,
2011-10-31 23:21:18,834 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
master. Retrying. Error was:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:2
06)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClien
t.java:311)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy3.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at
org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.j
ava:1443)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRe
gionServer.java:737)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:58
6)
at java.lang.Thread.run(Thread.java:619)
Also in the M4 Master log I am seeing,
2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 60010
webServer.getConnectors()[0].getLocalPort() returned 60010
2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 60010
2011-10-31 23:20:47,671 INFO org.mortbay.log: jetty-6.1.26
2011-10-31 23:20:48,691 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:60010
2011-10-31 23:20:48,691 DEBUG org.apache.hadoop.hbase.master.HMaster:
Started service threads
2011-10-31 23:20:50,192 INFO org.apache.hadoop.hbase.master.ServerManager:
Waiting on regionserver(s) to checkin
2011-10-31 23:20:51,695 INFO org.apache.hadoop.hbase.master.ServerManager:
Waiting on regionserver(s) to checkin
2011-10-31 23:20:53,197 INFO org.apache.hadoop.hbase.master.ServerManager:
Waiting on regionserver(s) to checkin
Seems the region servers are only communicating to the M1 Hmaster and not to
the new one!!
Any help regarding this is highly appreciated :handshake:
--
View this message in context:
http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32757197.html
Sent from the HBase User mailing list archive at Nabble.com.