You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hbase.apache.org by lilibiao2014 <li...@126.com> on 2014/06/26 10:55:28 UTC

all regionservers : numberOfOnlineRegions=0

Hey guys,

Yesterday our Hbase cluster had 4 of 11 regionserver don't work well, that
the numberOfOnlineRegions= 0 .
And when we restart the cluster,not only 4 but all of our regionservers this
occurs.
Here is the hbase master's log.Except the exception of the log ,we also find
few zookeeper's exception and log splitting exception.We can't find the real
cause.

Hope that helps and forgive my poor English : ) 
Thanks
Lee

2014-06-26 16:00:20,220 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:21,220 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:22,220 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:23,222 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:24,222 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:25,224 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:26,224 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:27,225 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:28,227 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:29,227 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:30,229 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:31,231 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:32,231 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:33,233 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:34,234 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:35,235 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:36,235 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:37,235 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:38,237 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:39,238 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:40,238 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:41,240 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:42,240 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:43,241 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:44,243 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:45,244 INFO org.apache.hadoop.hbase.master.SplitLogManager:
Skipping resubmissions of task
/hbase/splitlog/hdfs%3A%2F%2Fjyw-o-hadoop00%3A9000%2Fhbase%2F.logs%2Fjyw-o-h
adoop05.light.soufun.com%2C60020%2C1403110267692-splitting%2Fjyw-o-hadoop05.
light.soufun.com%252C60020%252C1403110267692.1403720357924 because threshold
3 reached
2014-06-26 16:00:45,244 INFO org.apache.hadoop.hbase.master.SplitLogManager:
Skipping resubmissions of task
/hbase/splitlog/hdfs%3A%2F%2Fjyw-o-hadoop00%3A9000%2Fhbase%2F.logs%2Fjyw-o-h
adoop05.light.soufun.com%2C60020%2C1403110267692-splitting%2Fjyw-o-hadoop05.
light.soufun.com%252C60020%252C1403110267692.1403723402640 because threshold
3 reached
2014-06-26 16:00:45,244 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:46,245 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:46,972 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
listener on 60000: readAndProcess threw exception java.io.IOException:
Connection reset by peer. Count of bytes read: 0
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcher.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
	at sun.nio.ch.IOUtil.read(IOUtil.java:171)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
	at
org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796)
	at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServe
r.java:1179)
	at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748
)
	at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServe
r.java:539)
	at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
:514)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
	at java.lang.Thread.run(Thread.java:662)

Re: all regionservers : numberOfOnlineRegions=0

Posted by Ted Yu <yu...@gmail.com>.

Which hbase release do you use ?
Have you checked region server log to see why log splitting had issues ?

Cheers

On Jun 26, 2014, at 1:55 AM, "lilibiao2014" <li...@126.com> wrote:

> Hey guys,
> 
> Yesterday our Hbase cluster had 4 of 11 regionserver don't work well, that
> the numberOfOnlineRegions= 0 .
> And when we restart the cluster,not only 4 but all of our regionservers this
> occurs.
> Here is the hbase master's log.Except the exception of the log ,we also find
> few zookeeper's exception and log splitting exception.We can't find the real
> cause.
> 
> Hope that helps and forgive my poor English : ) 
> Thanks
> Lee
> 
> 2014-06-26 16:00:20,220 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:21,220 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:22,220 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:23,222 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:24,222 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:25,224 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:26,224 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:27,225 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:28,227 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:29,227 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:30,229 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:31,231 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:32,231 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:33,233 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:34,234 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:35,235 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:36,235 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:37,235 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:38,237 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:39,238 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:40,238 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:41,240 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:42,240 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:43,241 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:44,243 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:45,244 INFO org.apache.hadoop.hbase.master.SplitLogManager:
> Skipping resubmissions of task
> /hbase/splitlog/hdfs%3A%2F%2Fjyw-o-hadoop00%3A9000%2Fhbase%2F.logs%2Fjyw-o-h
> adoop05.light.soufun.com%2C60020%2C1403110267692-splitting%2Fjyw-o-hadoop05.
> light.soufun.com%252C60020%252C1403110267692.1403720357924 because threshold
> 3 reached
> 2014-06-26 16:00:45,244 INFO org.apache.hadoop.hbase.master.SplitLogManager:
> Skipping resubmissions of task
> /hbase/splitlog/hdfs%3A%2F%2Fjyw-o-hadoop00%3A9000%2Fhbase%2F.logs%2Fjyw-o-h
> adoop05.light.soufun.com%2C60020%2C1403110267692-splitting%2Fjyw-o-hadoop05.
> light.soufun.com%252C60020%252C1403110267692.1403723402640 because threshold
> 3 reached
> 2014-06-26 16:00:45,244 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:46,245 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
> 0
> 2014-06-26 16:00:46,972 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
> listener on 60000: readAndProcess threw exception java.io.IOException:
> Connection reset by peer. Count of bytes read: 0
> java.io.IOException: Connection reset by peer
>    at sun.nio.ch.FileDispatcher.read0(Native Method)
>    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
>    at sun.nio.ch.IOUtil.read(IOUtil.java:171)
>    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
>    at
> org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796)
>    at
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServe
> r.java:1179)
>    at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748
> )
>    at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServe
> r.java:539)
>    at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
> :514)
>    at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>    at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>    at java.lang.Thread.run(Thread.java:662)
> 
>