You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by William Kang <we...@gmail.com> on 2010/03/08 06:12:41 UTC
HBase no response problem after inactivity
Hi,
I am running HBase 0.20.3 in a Pseudo-distributed mode. It was running fine.
But after about 10 hours' inactivities, I started to have trouble to connect
the HBase master. I cannot even shut it down. The HDFS is running fine. Any
idea what caused this? Many thanks.
William
Re: HBase no response problem after inactivity
Posted by Stack <st...@duboce.net>.
Did your master start? Check the master logs under $HBASE_HOME/logs/.
St.Ack
On Mon, Mar 8, 2010 at 5:56 PM, William Kang <we...@gmail.com> wrote:
> Hi,
> I got the same problem again today. So, I restarted the master process and
> got the following log:
>
> 2010-03-08 20:45:16,572 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: getMaster
> attempt 8 of 10 failed; retrying after sleep of 16000
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:831)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:712)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
> at $Proxy0.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:489)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:465)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:512)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:341)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1258)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1282)
> 2010-03-08 20:45:32,574 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/master
> got 127.0.1.1:60000
> 2010-03-08 20:45:32,575 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: getMaster
> attempt 9 of 10 failed; no more retrying.
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:831)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:712)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
> at $Proxy0.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:489)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:465)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:512)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:341)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1258)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1282)
> 2010-03-08 20:45:32,576 ERROR org.apache.hadoop.hbase.master.HMaster: master
> is not running
>
> If the whole log would help, please let me know.
> Many thanks.
>
>
> William
>
> On Mon, Mar 8, 2010 at 12:36 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> We see stuff like that when people try to put their laptop to sleep or
>> some other times people don't change the default data directory (which
>> is /tmp) and it gets somehow cleared by the OS.
>>
>> Anyways, even if the NN was formated, the logs are kept in your hbase
>> root's logs directory so unless you deleted that too it's supposed to
>> be there.
>>
>> J-D
>>
>> On Sun, Mar 7, 2010 at 9:31 PM, William Kang <we...@gmail.com>
>> wrote:
>> > Hi J-D,
>> > Thanks for reply so fast. Well, it is too late. I have format the
>> namenode.
>> > I will dump it next time. I assumed this should happen before.
>> >
>> >
>> > William
>> >
>> > On Mon, Mar 8, 2010 at 12:24 AM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>> >
>> >> And if you take a look at it's log, do you see Exceptions starting to
>> >> pop up at some point?
>> >>
>> >> J-D
>> >>
>> >> On Sun, Mar 7, 2010 at 9:12 PM, William Kang <we...@gmail.com>
>> >> wrote:
>> >> > Hi,
>> >> > I am running HBase 0.20.3 in a Pseudo-distributed mode. It was running
>> >> fine.
>> >> > But after about 10 hours' inactivities, I started to have trouble to
>> >> connect
>> >> > the HBase master. I cannot even shut it down. The HDFS is running
>> fine.
>> >> Any
>> >> > idea what caused this? Many thanks.
>> >> >
>> >> >
>> >> > William
>> >> >
>> >>
>> >
>>
>
Re: HBase no response problem after inactivity
Posted by William Kang <we...@gmail.com>.
Hi,
I got the same problem again today. So, I restarted the master process and
got the following log:
2010-03-08 20:45:16,572 INFO
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: getMaster
attempt 8 of 10 failed; retrying after sleep of 16000
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:831)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:712)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:489)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:465)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:512)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:341)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1258)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1282)
2010-03-08 20:45:32,574 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/master
got 127.0.1.1:60000
2010-03-08 20:45:32,575 INFO
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: getMaster
attempt 9 of 10 failed; no more retrying.
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:831)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:712)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:489)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:465)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:512)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:341)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1258)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1282)
2010-03-08 20:45:32,576 ERROR org.apache.hadoop.hbase.master.HMaster: master
is not running
If the whole log would help, please let me know.
Many thanks.
William
On Mon, Mar 8, 2010 at 12:36 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> We see stuff like that when people try to put their laptop to sleep or
> some other times people don't change the default data directory (which
> is /tmp) and it gets somehow cleared by the OS.
>
> Anyways, even if the NN was formated, the logs are kept in your hbase
> root's logs directory so unless you deleted that too it's supposed to
> be there.
>
> J-D
>
> On Sun, Mar 7, 2010 at 9:31 PM, William Kang <we...@gmail.com>
> wrote:
> > Hi J-D,
> > Thanks for reply so fast. Well, it is too late. I have format the
> namenode.
> > I will dump it next time. I assumed this should happen before.
> >
> >
> > William
> >
> > On Mon, Mar 8, 2010 at 12:24 AM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> And if you take a look at it's log, do you see Exceptions starting to
> >> pop up at some point?
> >>
> >> J-D
> >>
> >> On Sun, Mar 7, 2010 at 9:12 PM, William Kang <we...@gmail.com>
> >> wrote:
> >> > Hi,
> >> > I am running HBase 0.20.3 in a Pseudo-distributed mode. It was running
> >> fine.
> >> > But after about 10 hours' inactivities, I started to have trouble to
> >> connect
> >> > the HBase master. I cannot even shut it down. The HDFS is running
> fine.
> >> Any
> >> > idea what caused this? Many thanks.
> >> >
> >> >
> >> > William
> >> >
> >>
> >
>
Re: HBase no response problem after inactivity
Posted by Jean-Daniel Cryans <jd...@apache.org>.
We see stuff like that when people try to put their laptop to sleep or
some other times people don't change the default data directory (which
is /tmp) and it gets somehow cleared by the OS.
Anyways, even if the NN was formated, the logs are kept in your hbase
root's logs directory so unless you deleted that too it's supposed to
be there.
J-D
On Sun, Mar 7, 2010 at 9:31 PM, William Kang <we...@gmail.com> wrote:
> Hi J-D,
> Thanks for reply so fast. Well, it is too late. I have format the namenode.
> I will dump it next time. I assumed this should happen before.
>
>
> William
>
> On Mon, Mar 8, 2010 at 12:24 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> And if you take a look at it's log, do you see Exceptions starting to
>> pop up at some point?
>>
>> J-D
>>
>> On Sun, Mar 7, 2010 at 9:12 PM, William Kang <we...@gmail.com>
>> wrote:
>> > Hi,
>> > I am running HBase 0.20.3 in a Pseudo-distributed mode. It was running
>> fine.
>> > But after about 10 hours' inactivities, I started to have trouble to
>> connect
>> > the HBase master. I cannot even shut it down. The HDFS is running fine.
>> Any
>> > idea what caused this? Many thanks.
>> >
>> >
>> > William
>> >
>>
>
Re: HBase no response problem after inactivity
Posted by William Kang <we...@gmail.com>.
Hi J-D,
Thanks for reply so fast. Well, it is too late. I have format the namenode.
I will dump it next time. I assumed this should happen before.
William
On Mon, Mar 8, 2010 at 12:24 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> And if you take a look at it's log, do you see Exceptions starting to
> pop up at some point?
>
> J-D
>
> On Sun, Mar 7, 2010 at 9:12 PM, William Kang <we...@gmail.com>
> wrote:
> > Hi,
> > I am running HBase 0.20.3 in a Pseudo-distributed mode. It was running
> fine.
> > But after about 10 hours' inactivities, I started to have trouble to
> connect
> > the HBase master. I cannot even shut it down. The HDFS is running fine.
> Any
> > idea what caused this? Many thanks.
> >
> >
> > William
> >
>
Re: HBase no response problem after inactivity
Posted by Jean-Daniel Cryans <jd...@apache.org>.
And if you take a look at it's log, do you see Exceptions starting to
pop up at some point?
J-D
On Sun, Mar 7, 2010 at 9:12 PM, William Kang <we...@gmail.com> wrote:
> Hi,
> I am running HBase 0.20.3 in a Pseudo-distributed mode. It was running fine.
> But after about 10 hours' inactivities, I started to have trouble to connect
> the HBase master. I cannot even shut it down. The HDFS is running fine. Any
> idea what caused this? Many thanks.
>
>
> William
>