You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Matthew LeMieux <md...@mlogiciels.com> on 2010/09/01 08:53:36 UTC

Trying to restart HMaster

I just killed the HMaster hoping to get it to recover from region servers dying.   I'm getting a continuous stream these messages in the log file. 

Will the master ever stop with these messages, or is this another crash?

-Matthew

2010-09-01 06:47:14,267 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000, call regionServerReport(serverName=domU-12-31-39-18-1E-28.compute-1.internal,60020,1283315889121, load=(requests=0, regions=58, usedHeap=240, maxHeap=7995), [Lorg.apache.hadoop.hbase.HMsg;@254ba9a2, [Lorg.apache.hadoop.hbase.HRegionInfo;@a6dddcf) from 10.104.61.218:41736: error: java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close, pending open or closing. State: name=user_name_index,,1282242158507.8c9a40b89ee92e4b2f285b306a2d30ed., state=UNASSIGNED
java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close, pending open or closing. State: name=user_name_index,,1282242158507.8c9a40b89ee92e4b2f285b306a2d30ed., state=UNASSIGNED
        at org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1613)
        at org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:1101)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:662)
        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:467)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:425)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:335)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:684)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919)

Re: Trying to restart HMaster

Posted by Jean-Daniel Cryans <jd...@apache.org>.
Very hard to tell how it got there by just looking at the end result,
but you could try using the shell tools like disable_region then
close_region, and then enable_region on
user_name_index,,1282242158507.8c9a40b89ee92e4b2f285b306a2d30ed.

Also you could giving a spin to the the latest 0.89 release candidate
which I posted here
http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/

J-D

On Tue, Aug 31, 2010 at 11:53 PM, Matthew LeMieux <md...@mlogiciels.com> wrote:
> I just killed the HMaster hoping to get it to recover from region servers dying.   I'm getting a continuous stream these messages in the log file.
>
> Will the master ever stop with these messages, or is this another crash?
>
> -Matthew
>
> 2010-09-01 06:47:14,267 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000, call regionServerReport(serverName=domU-12-31-39-18-1E-28.compute-1.internal,60020,1283315889121, load=(requests=0, regions=58, usedHeap=240, maxHeap=7995), [Lorg.apache.hadoop.hbase.HMsg;@254ba9a2, [Lorg.apache.hadoop.hbase.HRegionInfo;@a6dddcf) from 10.104.61.218:41736: error: java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close, pending open or closing. State: name=user_name_index,,1282242158507.8c9a40b89ee92e4b2f285b306a2d30ed., state=UNASSIGNED
> java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be closed if it was not already marked as pending close, pending open or closing. State: name=user_name_index,,1282242158507.8c9a40b89ee92e4b2f285b306a2d30ed., state=UNASSIGNED
>        at org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1613)
>        at org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:1101)
>        at org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:662)
>        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:467)
>        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:425)
>        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:335)
>        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:684)
>        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919)
>
>