You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ron van der Vegt <ro...@openindex.io> on 2014/09/01 10:47:58 UTC
AW: Could not get distributed Hbase stable, stopping every ~24 hours.
I have checked all bullets but the hbase cluster is still crashing:
*
Make sure you give plenty of RAM (in hbase-env.sh), the default of 1GB won't be able to sustain long running imports.
This is set to the default 1GB, this should be more than enough for the amount of data that is stored. I also checked if there is a memory leak, but that is not the case.
*
Make sure you don't swap, the JVM never behaves well under swapping.
Swapping is disabled.
*
Make sure you are not CPU starving the RegionServer thread. For example, if you are running a MapReduce job using 6 CPU-intensive tasks on a machine with 4 cores, you are probably starving the RegionServer enough to create longer garbage collection pauses.
This is not the case.
*
Increase the ZooKeeper session timeout
Increased it to the suggested 120 seconds, but still get the same errors.
Do you have any other suggesting?
Thanks in advice,
Ron
-----Oorspronkelijk bericht-----
> Afzender:Ted Yu <yuzhihong@gmail.com <ma...@gmail.com> >
> Verstuurd: Dinsdag 26 Augustus 2014 11:27
> Aan: user@hbase.apache.org <ma...@hbase.apache.org>
> Onderwerp: Re: Could not get distributed Hbase stable, stopping every ˜24 hours.
>
> Please take a look at 15.9.2.7 under
> http://hbase.apache.org/book.html#trouble.rs.runtime <http://hbase.apache.org/book.html#trouble.rs.runtime>
>
> Cheers
>
>
> On Tue, Aug 26, 2014 at 1:32 AM, Ron van der Vegt <
> ron.van.der.vegt@openindex.io <ma...@openindex.io> > wrote:
>
> > First off all thanks for your reply. We are using three zookeeper server.
> > All with version 3.4.6.
> > Below the logs of one of the zookeeper server. It seems that the
> > connection will at some time just drop.
> >
> > In the meantime I will monitor heap dumps to check if there is a memory
> > leakage. But my feeling tells me the memory leakage is not the cause but
> > more the effect.
> > Do you see anything in all those logs which we could determine what is
> > happening?
> >
> > Thanks!
> >
> > 2014-08-23 07:00:19,927 [myid:3] - INFO
> > [SessionTracker:ZooKeeperServer@347] - Expiring session
> > 0x347fc15265a00ec, timeout of 40000ms exceeded
> > 2014-08-23 07:00:19,927 [myid:3] - INFO
> > [SessionTracker:ZooKeeperServer@347] - Expiring session
> > 0x147fc1616d200be, timeout of 40000ms exceeded
> > 2014-08-23 07:00:19,927 [myid:3] - INFO
> > [SessionTracker:ZooKeeperServer@347] - Expiring session
> > 0x247fc16c80500d6, timeout of 40000ms exceeded
> > 2014-08-23 07:00:19,928 [myid:3] - INFO
> > [SessionTracker:ZooKeeperServer@347] - Expiring session
> > 0x147fc1616d200bb, timeout of 40000ms exceeded
> > 2014-08-23 07:00:19,928 [myid:3] - INFO
> > [SessionTracker:ZooKeeperServer@347] - Expiring session
> > 0x347fc15265a00eb, timeout of 40000ms exceeded
> > 2014-08-23 07:00:19,928 [myid:3] - INFO
> > [SessionTracker:ZooKeeperServer@347] - Expiring session
> > 0x147fc1616d200bd, timeout of 40000ms exceeded
> > 2014-08-23 07:00:19,928 [myid:3] - INFO
> > [SessionTracker:ZooKeeperServer@347] - Expiring session
> > 0x347fc15265a00ea, timeout of 40000ms exceeded
> > 2014-08-23 07:00:19,928 [myid:3] - INFO
> > [SessionTracker:ZooKeeperServer@347] - Expiring session
> > 0x347fc15265a00ef, timeout of 40000ms exceeded
> > 2014-08-23 07:00:19,928 [myid:3] - INFO
> > [SessionTracker:ZooKeeperServer@347] - Expiring session
> > 0x247ffe833880000, timeout of 40000ms exceeded
> > 2014-08-23 07:00:19,929 [myid:3] - INFO
> > [SessionTracker:ZooKeeperServer@347] - Expiring session
> > 0x247ffe833880001, timeout of 40000ms exceeded
> > 2014-08-23 07:00:19,929 [myid:3] - INFO
> > [SessionTracker:ZooKeeperServer@347] - Expiring session
> > 0x247fc16c80500d2, timeout of 40000ms exceeded
> > 2014-08-23 07:00:19,929 [myid:3] - INFO
> > [SessionTracker:ZooKeeperServer@347] - Expiring session
> > 0x247fc16c80500d5, timeout of 40000ms exceeded
> > 2014-08-23 07:00:19,930 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x347fc15265a00ec
> > 2014-08-23 07:00:19,930 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x147fc1616d200be
> > 2014-08-23 07:00:19,930 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x247fc16c80500d6
> > 2014-08-23 07:00:19,930 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x147fc1616d200bb
> > 2014-08-23 07:00:19,931 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x347fc15265a00eb
> > 2014-08-23 07:00:19,931 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x147fc1616d200bd
> > 2014-08-23 07:00:19,931 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x347fc15265a00ea
> > 2014-08-23 07:00:19,931 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x347fc15265a00ef
> > 2014-08-23 07:00:19,935 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x247ffe833880000
> > 2014-08-23 07:00:19,936 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x247ffe833880001
> > 2014-08-23 07:00:19,936 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x247fc16c80500d2
> > 2014-08-23 07:00:19,937 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x247fc16c80500d5
> > 2014-08-23 07:00:19,984 [myid:3] - INFO
> > [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> > client /141.105.120.152:52656 which had sessionid 0x347fc15265a00ec
> > 2014-08-23 07:00:19,985 [myid:3] - INFO
> > [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> > client /141.105.120.152:52651 which had sessionid 0x147fc1616d200be
> > 2014-08-23 07:00:19,986 [myid:3] - INFO
> > [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> > client /141.105.120.150:44649 which had sessionid 0x247fc16c80500d6
> > 2014-08-23 07:00:19,990 [myid:3] - INFO
> > [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> > client /141.105.120.153:60445 which had sessionid 0x147fc1616d200bb
> > 2014-08-23 07:00:19,993 [myid:3] - INFO
> > [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> > client /141.105.120.154:47997 which had sessionid 0x347fc15265a00eb
> > 2014-08-23 07:00:19,995 [myid:3] - INFO
> > [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> > client /141.105.120.154:47999 which had sessionid 0x147fc1616d200bd
> > 2014-08-23 07:00:19,997 [myid:3] - INFO
> > [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> > client /141.105.120.151:34773 which had sessionid 0x347fc15265a00ea
> > 2014-08-23 07:00:19,998 [myid:3] - INFO
> > [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> > client /141.105.120.153:60444 which had sessionid 0x347fc15265a00ef
> > 2014-08-23 07:00:20,000 [myid:3] - INFO
> > [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> > client /141.105.120.102:37154 which had sessionid 0x247ffe833880000
> > 2014-08-23 07:00:20,002 [myid:3] - INFO
> > [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> > client /141.105.120.102:37155 which had sessionid 0x247ffe833880001
> > 2014-08-23 07:00:20,005 [myid:3] - INFO
> > [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> > client /141.105.120.150:48086 which had sessionid 0x247fc16c80500d2
> > 2014-08-23 07:00:20,006 [myid:3] - WARN [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@218] - Ignoring unexpected
> > runtime exception
> > java.nio.channels.CancelledKeyException
> > at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
> > at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
> > at
> > org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
> > at java.lang.Thread.run(Thread.java:744)
> > 2014-08-23 07:00:20,008 [myid:3] - INFO
> > [CommitProcessor:3:NIOServerCnxn@1007] - Closed socket connection for
> > client /141.105.120.151:38066 which had sessionid 0x247fc16c80500d5
> > 2014-08-23 07:00:20,008 [myid:3] - WARN [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@218] - Ignoring unexpected
> > runtime exception
> > java.nio.channels.CancelledKeyException
> > at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
> > at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
> > at
> > org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
> > at java.lang.Thread.run(Thread.java:744)
> > 2014-08-23 07:00:34,185 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> > connection from /141.105.120.102:37423
> > 2014-08-23 07:00:34,185 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> > establish new session at /141.105.120.102:37423
> > 2014-08-23 07:00:34,189 [myid:3] - INFO
> > [CommitProcessor:3:ZooKeeperServer@617] - Established session
> > 0x347fcb5a0130001 with negotiated timeout 40000 for client /
> > 141.105.120.102:37423
> > 2014-08-23 07:00:34,196 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException
> > when processing sessionid:0x347fcb5a0130001 type:delete cxid:0x5
> > zxid:0x6c0000001f txntype:-1 reqpath:n/a Error Path:/hbase/backup-masters/
> > vps2008.directvps.nl,60000,1408691163492 Error:KeeperErrorCode = NoNode
> > for /hbase/backup-masters/vps2008.directvps.nl,60000,1408691163492
> > 2014-08-23 07:00:34,409 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException
> > when processing sessionid:0x347fcb5a0130001 type:create cxid:0x12
> > zxid:0x6c00000021 txntype:-1 reqpath:n/a Error
> > Path:/hbase/online-snapshot/acquired Error:KeeperErrorCode = NodeExists for
> > /hbase/online-snapshot/acquired
> > 2014-08-23 07:00:45,466 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x24801068aff0001
> > 2014-08-23 07:00:45,875 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> > connection from /178.21.116.224:58692
> > 2014-08-23 07:00:45,875 [myid:3] - WARN [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> > client /178.21.116.224:58692; will be dropped if server is in r-o mode
> > 2014-08-23 07:00:45,875 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> > establish new session at /178.21.116.224:58692
> > 2014-08-23 07:00:45,884 [myid:3] - INFO
> > [CommitProcessor:3:ZooKeeperServer@617] - Established session
> > 0x347fcb5a0130002 with negotiated timeout 5000 for client /
> > 178.21.116.224:58692
> > 2014-08-23 07:00:45,887 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x347fcb5a0130002
> > 2014-08-23 07:00:45,891 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> > client /178.21.116.224:58692 which had sessionid 0x347fcb5a0130002
> > 2014-08-23 07:00:46,494 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> > connection from /178.21.116.224:58694
> > 2014-08-23 07:00:46,494 [myid:3] - WARN [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> > client /178.21.116.224:58694; will be dropped if server is in r-o mode
> > 2014-08-23 07:00:46,494 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> > establish new session at /178.21.116.224:58694
> > 2014-08-23 07:00:46,498 [myid:3] - INFO
> > [CommitProcessor:3:ZooKeeperServer@617] - Established session
> > 0x347fcb5a0130003 with negotiated timeout 5000 for client /
> > 178.21.116.224:58694
> > 2014-08-23 07:00:46,501 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x347fcb5a0130003
> > 2014-08-23 07:00:46,503 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> > client /178.21.116.224:58694 which had sessionid 0x347fcb5a0130003
> > 2014-08-23 07:00:47,314 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> > connection from /178.21.116.224:58696
> > 2014-08-23 07:00:47,314 [myid:3] - WARN [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> > client /178.21.116.224:58696; will be dropped if server is in r-o mode
> > 2014-08-23 07:00:47,315 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> > establish new session at /178.21.116.224:58696
> > 2014-08-23 07:00:47,318 [myid:3] - INFO
> > [CommitProcessor:3:ZooKeeperServer@617] - Established session
> > 0x347fcb5a0130004 with negotiated timeout 5000 for client /
> > 178.21.116.224:58696
> > 2014-08-23 07:00:47,321 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x347fcb5a0130004
> > 2014-08-23 07:00:47,324 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> > client /178.21.116.224:58696 which had sessionid 0x347fcb5a0130004
> > 2014-08-23 07:00:48,334 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> > connection from /178.21.116.224:58698
> > 2014-08-23 07:00:48,335 [myid:3] - WARN [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> > client /178.21.116.224:58698; will be dropped if server is in r-o mode
> > 2014-08-23 07:00:48,335 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> > establish new session at /178.21.116.224:58698
> > 2014-08-23 07:00:48,338 [myid:3] - INFO
> > [CommitProcessor:3:ZooKeeperServer@617] - Established session
> > 0x347fcb5a0130005 with negotiated timeout 5000 for client /
> > 178.21.116.224:58698
> > 2014-08-23 07:00:48,342 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x347fcb5a0130005
> > 2014-08-23 07:00:48,345 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> > client /178.21.116.224:58698 which had sessionid 0x347fcb5a0130005
> > 2014-08-23 07:00:49,399 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x148010689410000
> > 2014-08-23 07:00:50,459 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x24801068aff0002
> > 2014-08-23 07:00:51,557 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x24801068aff0003
> > 2014-08-23 07:00:51,563 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> > connection from /178.21.116.224:58706
> > 2014-08-23 07:00:51,563 [myid:3] - WARN [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> > client /178.21.116.224:58706; will be dropped if server is in r-o mode
> > 2014-08-23 07:00:51,563 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> > establish new session at /178.21.116.224:58706
> > 2014-08-23 07:00:51,568 [myid:3] - INFO
> > [CommitProcessor:3:ZooKeeperServer@617] - Established session
> > 0x347fcb5a0130006 with negotiated timeout 5000 for client /
> > 178.21.116.224:58706
> > 2014-08-23 07:00:51,571 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x347fcb5a0130006
> > 2014-08-23 07:00:51,575 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> > client /178.21.116.224:58706 which had sessionid 0x347fcb5a0130006
> > 2014-08-23 07:00:52,855 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x24801068aff0004
> > 2014-08-23 07:00:52,870 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x24801068aff0005
> > 2014-08-23 07:00:54,914 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> > connection from /178.21.116.224:58712
> > 2014-08-23 07:00:54,914 [myid:3] - WARN [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> > client /178.21.116.224:58712; will be dropped if server is in r-o mode
> > 2014-08-23 07:00:54,915 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> > establish new session at /178.21.116.224:58712
> > 2014-08-23 07:00:54,918 [myid:3] - INFO
> > [CommitProcessor:3:ZooKeeperServer@617] - Established session
> > 0x347fcb5a0130007 with negotiated timeout 5000 for client /
> > 178.21.116.224:58712
> > 2014-08-23 07:00:54,921 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x347fcb5a0130007
> > 2014-08-23 07:00:54,925 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> > client /178.21.116.224:58712 which had sessionid 0x347fcb5a0130007
> > 2014-08-23 07:00:55,357 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x148010689410001
> > 2014-08-23 07:00:55,997 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x148010689410002
> > 2014-08-23 07:00:56,018 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x24801068aff0006
> > 2014-08-23 07:00:57,047 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x24801068aff0007
> > 2014-08-23 07:00:58,583 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x148010689410003
> > 2014-08-23 07:00:59,642 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x24801068aff0008
> > 2014-08-23 07:01:00,737 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x148010689410004
> > 2014-08-23 07:01:01,904 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x24801068aff0009
> > 2014-08-23 07:01:08,186 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> > connection from /178.21.116.224:58730
> > 2014-08-23 07:01:08,187 [myid:3] - WARN [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old
> > client /178.21.116.224:58730; will be dropped if server is in r-o mode
> > 2014-08-23 07:01:08,187 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> > establish new session at /178.21.116.224:58730
> > 2014-08-23 07:01:08,191 [myid:3] - INFO
> > [CommitProcessor:3:ZooKeeperServer@617] - Established session
> > 0x347fcb5a0130008 with negotiated timeout 5000 for client /
> > 178.21.116.224:58730
> > 2014-08-23 07:01:08,194 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x347fcb5a0130008
> > 2014-08-23 07:01:08,197 [myid:3] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> > client /178.21.116.224:58730 which had sessionid 0x347fcb5a0130008
> > 2014-08-23 07:01:09,735 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x24801068aff000a
> > 2014-08-23 07:01:11,799 [myid:3] - INFO [ProcessThread(sid:3
> > cport:-1)::PrepRequestProcessor@494] - Processed session termination for
> > sessionid: 0x24801068aff000b
> >
> >
> >
> > -----Oorspronkelijk bericht-----
> > > Afzender:Ted Yu <yuzhihong@gmail.com <ma...@gmail.com> <mailto:yuzhihong@gmail.com <ma...@gmail.com> > >
> > > Verstuurd: Maandag 25 Augustus 2014 16:45
> > > Aan: user@hbase.apache.org <ma...@hbase.apache.org> <mailto:user@hbase.apache.org <ma...@hbase.apache.org> >
> > > Onderwerp: Re: Could not get distributed Hbase stable, stopping every
> > ˜24 hours.
> > >
> > > Have you checked zookeeper logs ?
> > > What zookeeper release are you using ?
> > >
> > > bq. Could there be a memory leakage?
> > >
> > > You can use jmap to capture heap memory details:
> > >
> > > http://docs.oracle.com/javase/6/docs/technotes/tools/share/jmap.html <http://docs.oracle.com/javase/6/docs/technotes/tools/share/jmap.html> <
> > " target="_blank">http://docs.oracle.com/javase/6/docs/technotes/tools/share/jmap.html> <http://docs.oracle.com/javase/6/docs/technotes/tools/share/jmap.html> ;
> > >
> > > Cheers
> > >
> > >
> > > On Mon, Aug 25, 2014 at 4:51 AM, Ron van der Vegt <
> > > ron.van.der.vegt@openindex.io <ma...@openindex.io> <mailto:ron.van.der.vegt@openindex.io <ma...@openindex.io> > >
> > wrote:
> > >
> > > > Hi all!
> > > >
> > > > I have setup one master and 5 regionservers to collect log data. But
> > every
> > > > ˜24 hours, at random times, the regionservers generating a fatal error
> > and
> > > > all stopping one by one. Eventually the master will stop. I also see
> > some
> > > > weird characters before the server names in the logs. Seems like some
> > > > encoding issue.
> > > >
> > > > I have read in the documentation, that if the garbage collection is
> > taking
> > > > to long, you will also get the session expired message. But I have
> > logged
> > > > the GC on the master, and it seems oke. Could someone help me figure
> > out
> > > > why this is happening?
> > > >
> > > > Furthermore, I am currently monitoring the memory usage of the master
> > with
> > > > JMX. I notice that the heap size is slowly growing. Could there be a
> > memory
> > > > leakage?
> > > > xmx is set to 1gb.
> > > >
> > > > Setup:
> > > > hbase 0.94.20
> > > > hadoop 1.2.1
> > > > debian wheezy
> > > >
> > > > Thanks in advice,
> > > >
> > > > Ron
> > > >
> > > > Logs of master:
> > > > ===============
> > > >
> > > > 2014-08-23 07:00:20,104 WARN
> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> > transient
> > > > ZooKeeper exception:
> > > > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > > > KeeperErrorCode = ConnectionLoss for /hbase/unassigned/70236052
> > > > 2014-08-23 07:00:20,406 ERROR org.apache.hadoop.hbase.master.HMaster:
> > > > Region server vps2060.directvps.nl,60020,1408691165501 reported a
> > fatal
> > > > error:
> > > > ABORTING region server vps2060.directvps.nl,60020,1408691165501:
> > > >
> > regionserver:60020-0x347fc15265a00eb-0x347fc15265a00eb-0x347fc15265a00eb
> > > >
> > regionserver:60020-0x347fc15265a00eb-0x347fc15265a00eb-0x347fc15265a00eb
> > > > received expired from ZooKeeper, aborting
> > > > Cause:
> > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > KeeperErrorCode = Session expired
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:384)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
> > > > at
> > > >
> > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > > > at
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > >
> > > > 2014-08-23 07:00:20,911 ERROR org.apache.hadoop.hbase.master.HMaster:
> > > > Region server vps2057.directvps.nl,60020,1408691165499 reported a
> > fatal
> > > > error:
> > > > ABORTING region server vps2057.directvps.nl,60020,1408691165499:
> > > >
> > regionserver:60020-0x347fc15265a00ea-0x347fc15265a00ea-0x347fc15265a00ea-0x347fc15265a00ea
> > > >
> > regionserver:60020-0x347fc15265a00ea-0x347fc15265a00ea-0x347fc15265a00ea-0x347fc15265a00ea
> > > > received expired from ZooKeeper, aborting
> > > > Cause:
> > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > KeeperErrorCode = Session expired
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:384)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
> > > > at
> > > >
> > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > > > at
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > >
> > > > 2014-08-23 07:00:21,001 ERROR org.apache.hadoop.hbase.master.HMaster:
> > > > Region server vps2059.directvps.nl,60020,1408691165851 reported a
> > fatal
> > > > error:
> > > > ABORTING region server vps2059.directvps.nl,60020,1408691165851:
> > > >
> > regionserver:60020-0x147fc1616d200bb-0x147fc1616d200bb-0x147fc1616d200bb
> > > >
> > regionserver:60020-0x147fc1616d200bb-0x147fc1616d200bb-0x147fc1616d200bb
> > > > received expired from ZooKeeper, aborting
> > > > Cause:
> > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > KeeperErrorCode = Session expired
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:384)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
> > > > at
> > > >
> > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > > > at
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > >
> > > > 2014-08-23 07:00:21,056 ERROR org.apache.hadoop.hbase.master.HMaster:
> > > > Region server vps2058.directvps.nl,60020,1408691165675 reported a
> > fatal
> > > > error:
> > > > ABORTING region server vps2058.directvps.nl,60020,1408691165675:
> > > >
> > regionserver:60020-0x347fc15265a00ec-0x347fc15265a00ec-0x347fc15265a00ec-0x347fc15265a00ec
> > > >
> > regionserver:60020-0x347fc15265a00ec-0x347fc15265a00ec-0x347fc15265a00ec-0x347fc15265a00ec
> > > > received expired from ZooKeeper, aborting
> > > > Cause:
> > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > KeeperErrorCode = Session expired
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:384)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
> > > > at
> > > >
> > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > > > at
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > >
> > > > 2014-08-23 07:00:22,140 WARN
> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> > transient
> > > > ZooKeeper exception:
> > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > KeeperErrorCode = Session expired for /hbase/unassigned/70236052
> > > > 2014-08-23 07:00:26,141 WARN
> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> > transient
> > > > ZooKeeper exception:
> > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > KeeperErrorCode = Session expired for /hbase/unassigned/70236052
> > > > 2014-08-23 07:00:34,114 ERROR org.apache.hadoop.hbase.master.HMaster:
> > > > Region server vps2056.directvps.nl,60020,1408691165439 reported a
> > fatal
> > > > error:
> > > > ABORTING region server vps2056.directvps.nl,60020,1408691165439:
> > > > Unexpected exception handling nodeDeleted event
> > > > Cause:
> > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > KeeperErrorCode = Session expired for /hbase/master
> > > > at
> > > > org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> > > > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:420)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:182)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:318)
> > > > at
> > > >
> > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > > > at
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > >
> > > > 2014-08-23 07:00:34,118 ERROR org.apache.hadoop.hbase.master.HMaster:
> > > > Region server vps2056.directvps.nl,60020,1408691165439 reported a
> > fatal
> > > > error:
> > > > ABORTING region server vps2056.directvps.nl,60020,1408691165439:
> > > >
> > regionserver:60020-0x247fc16c80500d2-0x247fc16c80500d2-0x247fc16c80500d2-0x247fc16c80500d2-0x247fc16c80500d2
> > > >
> > regionserver:60020-0x247fc16c80500d2-0x247fc16c80500d2-0x247fc16c80500d2-0x247fc16c80500d2-0x247fc16c80500d2
> > > > received expired from ZooKeeper, aborting
> > > > Cause:
> > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > KeeperErrorCode = Session expired
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:384)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
> > > > at
> > > >
> > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > > > at
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > >
> > > > 2014-08-23 07:00:34,141 WARN
> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> > transient
> > > > ZooKeeper exception:
> > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > KeeperErrorCode = Session expired for /hbase/unassigned/70236052
> > > > 2014-08-23 07:00:34,142 ERROR
> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper
> > getData
> > > > failed after 3 retries
> > > > 2014-08-23 07:00:34,152 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> > > >
> > master:60000-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x347fcb5a0130000-0x247ffe833880001-0x247ffe833880001-0x247ffe833880001
> > > > Unable to get data of znode /hbase/unassigned/70236052
> > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > KeeperErrorCode = Session expired for /hbase/unassigned/70236052
> > > > at
> > > > org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> > > > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > > > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:709)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:685)
> > > > at
> > > > org.apache.hadoop.hbase.zookeeper.ZKAssign.getData(ZKAssign.java:852)
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.AssignmentManager.isCarryingRegion(AssignmentManager.java:3274)
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.AssignmentManager.isCarryingRoot(AssignmentManager.java:3255)
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.ServerManager.expireServer(ServerManager.java:382)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.RegionServerTracker.nodeDeleted(RegionServerTracker.java:122)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:318)
> > > > at
> > > >
> > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > > > at
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > > 2014-08-23 07:00:34,152 ERROR
> > > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher:
> > > >
> > master:60000-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x347fcb5a0130000-0x247ffe833880001-0x247ffe833880001-0x247ffe833880001
> > > > Received unexpected KeeperException, re-throwing exception
> > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > KeeperErrorCode = Session expired for /hbase/unassigned/70236052
> > > > at
> > > > org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> > > > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > > > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:709)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:685)
> > > > at
> > > > org.apache.hadoop.hbase.zookeeper.ZKAssign.getData(ZKAssign.java:852)
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.AssignmentManager.isCarryingRegion(AssignmentManager.java:3274)
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.AssignmentManager.isCarryingRoot(AssignmentManager.java:3255)
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.ServerManager.expireServer(ServerManager.java:382)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.RegionServerTracker.nodeDeleted(RegionServerTracker.java:122)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:318)
> > > > at
> > > >
> > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > > > at
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > > 2014-08-23 07:00:34,163 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > > Master server abort: loaded coprocessors are: []
> > > > 2014-08-23 07:00:34,215 WARN
> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> > > > /hbase/backup-masters/vps2008.directvps.nl,60000,1408691163492 already
> > > > deleted, and this is not a retry
> > > > 2014-08-23 07:05:34,165 WARN
> > > > org.apache.hadoop.hbase.master.SplitLogManager: Interrupted while
> > waiting
> > > > for log splits to be completed
> > > > 2014-08-23 07:05:34,179 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > > Unexpected ZK exception reading unassigned node for region=70236052
> > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > KeeperErrorCode = Session expired for /hbase/unassigned/70236052
> > > > at
> > > > org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> > > > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > > > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:709)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:685)
> > > > at
> > > > org.apache.hadoop.hbase.zookeeper.ZKAssign.getData(ZKAssign.java:852)
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.AssignmentManager.isCarryingRegion(AssignmentManager.java:3274)
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.AssignmentManager.isCarryingRoot(AssignmentManager.java:3255)
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.ServerManager.expireServer(ServerManager.java:382)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.RegionServerTracker.nodeDeleted(RegionServerTracker.java:122)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:318)
> > > > at
> > > >
> > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > > > at
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > > 2014-08-23 07:05:34,179 WARN
> > > > org.apache.hadoop.hbase.master.SplitLogManager: error while splitting
> > logs
> > > > in [hdfs://
> > > >
> > namenode.openindex.io:8020/hbase/.logs/vps2058.directvps.nl,60020,1408691165675-splitting
> > ]
> > > > installed = 2 but only 0 done
> > > > 2014-08-23 07:05:34,184 INFO org.apache.hadoop.ipc.HBaseServer:
> > Stopping
> > > > server on 60000
> > > > 2014-08-23 07:05:34,185 WARN
> > > > org.apache.hadoop.hbase.master.CatalogJanitor: Failed scan of catalog
> > table
> > > > java.io.IOException: Giving up after tries=1
> > > > at
> > > >
> > org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:210)
> > > > at
> > > >
> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:188)
> > > > at
> > > >
> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:82)
> > > > at
> > > >
> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:67)
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.CatalogJanitor.getSplitParents(CatalogJanitor.java:126)
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:137)
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:93)
> > > > at org.apache.hadoop.hbase.Chore.run(Chore.java:67)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > > Caused by: java.lang.InterruptedException: sleep interrupted
> > > > at java.lang.Thread.sleep(Native Method)
> > > > at
> > > >
> > org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:207)
> > > > ... 8 more
> > > > 2014-08-23 07:05:34,185 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > Server
> > > > handler 5 on 60000: exiting
> > > > 2014-08-23 07:05:34,185 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > Server
> > > > handler 6 on 60000: exiting
> > > > 2014-08-23 07:05:34,186 INFO org.apache.hadoop.ipc.HBaseServer: REPL
> > IPC
> > > > Server handler 0 on 60000: exiting
> > > > 2014-08-23 07:05:34,186 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > Server
> > > > handler 2 on 60000: exiting
> > > > 2014-08-23 07:05:34,186 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > Server
> > > > handler 3 on 60000: exiting
> > > > 2014-08-23 07:05:34,186 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > Server
> > > > handler 4 on 60000: exiting
> > > > 2014-08-23 07:05:34,185 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > Server
> > > > handler 7 on 60000: exiting
> > > > 2014-08-23 07:05:34,185 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > Server
> > > > handler 8 on 60000: exiting
> > > > 2014-08-23 07:05:34,213 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > Server
> > > > handler 9 on 60000: exiting
> > > > 2014-08-23 07:05:34,213 INFO org.apache.hadoop.ipc.HBaseServer: REPL
> > IPC
> > > > Server handler 2 on 60000: exiting
> > > > 2014-08-23 07:05:34,213 INFO org.apache.hadoop.ipc.HBaseServer:
> > Stopping
> > > > IPC Server listener on 60000
> > > > 2014-08-23 07:05:34,213 INFO org.apache.hadoop.ipc.HBaseServer:
> > Stopping
> > > > IPC Server Responder
> > > > 2014-08-23 07:05:34,214 INFO org.apache.hadoop.ipc.HBaseServer:
> > Stopping
> > > > IPC Server Responder
> > > > 2014-08-23 07:05:34,212 INFO org.apache.hadoop.ipc.HBaseServer: REPL
> > IPC
> > > > Server handler 1 on 60000: exiting
> > > > 2014-08-23 07:05:34,186 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > Server
> > > > handler 0 on 60000: exiting
> > > > 2014-08-23 07:05:34,185 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > Server
> > > > handler 1 on 60000: exiting
> > > > 2014-08-23 07:05:34,256 INFO org.mortbay.log: Stopped
> > > > SelectChannelConnector@0.0.0.0:60010
> > > > 2014-08-23 07:05:34,259 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > > Master server abort: loaded coprocessors are: []
> > > > 2014-08-23 07:05:34,260 FATAL org.apache.hadoop.hbase.master.HMaster:
> > > >
> > master:60000-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x347fcb5a0130000-0x247ffe833880001-0x247ffe833880001-0x247ffe833880001-0x347fcb5a0130001
> > > >
> > master:60000-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x147fc1616d200ba-0x347fcb5a0130000-0x247ffe833880001-0x247ffe833880001-0x247ffe833880001-0x347fcb5a0130001
> > > > received expired from ZooKeeper, aborting
> > > > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > > KeeperErrorCode = Session expired
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:384)
> > > > at
> > > >
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
> > > > at
> > > >
> > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> > > > at
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> > > > 2014-08-23 07:05:34,414 ERROR
> > > > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start
> > master
> > > > java.lang.RuntimeException: HMaster Aborted
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)
> > > > at
> > > >
> > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > > > at
> > > >
> > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
> > > > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2129)
> > > >
> > > > GC log:
> > > > =======
> > > >
> > > > 0.185: Application time: 0.1304320 seconds
> > > > 0.185: [GC0.185: [ParNew: 4288K->511K(4800K), 0.0120520 secs]
> > > > 4288K->1008K(15424K), 0.0121600 secs] [Times: user=0.01 sys=0.01,
> > real=0.01
> > > > secs]
> > > > 0.197: Total time for which application threads were stopped: 0.0126240
> > > > seconds
> > > > Heap
> > > > par new generation total 4800K, used 3580K [0x00000000b7200000,
> > > > 0x00000000b7730000, 0x00000000c1860000)
> > > > eden space 4288K, 71% used [0x00000000b7200000, 0x00000000b74ff328,
> > > > 0x00000000b7630000)
> > > > from space 512K, 99% used [0x00000000b76b0000, 0x00000000b772fff8,
> > > > 0x00000000b7730000)
> > > > to space 512K, 0% used [0x00000000b7630000, 0x00000000b7630000,
> > > > 0x00000000b76b0000)
> > > > concurrent mark-sweep generation total 10624K, used 496K
> > > > [0x00000000c1860000, 0x00000000c22c0000, 0x00000000f5a00000)
> > > > concurrent-mark-sweep perm gen total 21248K, used 6688K
> > > > [0x00000000f5a00000, 0x00000000f6ec0000, 0x0000000100000000)
> > > > 0.370: Application time: 0.1728650 seconds
> > > >
> > > >
> > >
> >
> >
>