You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by 梁景明 <fu...@gmail.com> on 2010/06/01 04:49:34 UTC
Re: how to recover region server
hi ,hadoop 0.20.1 and hbase 0.20.4
two nodes . yes, they work hard,
they also severed the web.
they runs ok for one month .
is there a way to rescue the regionserver?
2010/5/30 Stack <st...@duboce.net>
> Its hard to read logs out of an email and the snippet below is too
> short to tell for sure whats going on. I see this on the end 'Session
> Expired', which would seem to indicate that hbase lost its lease with
> zookeeper. Are these nodes working hard?
>
> Tell us more about your cluster. How many nodes, what versions of hbase.
>
> St.Ack
>
> On Thu, May 27, 2010 at 8:15 PM, 梁景明 <fu...@gmail.com> wrote:
> > hi , my hbase crashed , here is the log ,and i wanna recover my region
> > server ,after trying
> > "hbase/bin/hbase-daemon.sh start regionserver"
> >
> > it failed to start, what 's the problem to shutdown the region server?
> > and how can i recover my region server , i dont want to reinstall all
> region
> > server
> > recover hbase,then import my backup;
> >
> > does region server to be recovered?
> >
> > thanks for any help;
> >
> > 2010-05-28 07:57:19,646 DEBUG
> > org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
> > Total=0.9473572MB (993376), Free=198.41515MB (208053360), Max=199.3625MB
> > (209046736), Counts: Blocks=4, Access=16584, Hit=16578, Miss=6,
> Evictions=0,
> > Evicted=0, Ratios: Hit Ratio=99.96381998062134%, Miss
> > Ratio=0.03617944894358516%, Evicted/Run=NaN
> > 2010-05-28 07:57:19,651 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> > 3212049130314122049 lease expired
> > 2010-05-28 07:57:19,729 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> > 3787094744445092555 lease expired
> > 2010-05-28 07:57:19,729 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > org.apache.hadoop.hbase.LeaseException: lease '3212049130314122049' does
> not
> > exist or has already expired
> > at org.apache.hadoop.hbase.Leases.renewLease(Leases.java:190)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1899)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877)
> > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> > 2010-05-28 07:57:19,853 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > org.apache.hadoop.hbase.UnknownScannerException: Name:
> 3787094744445092555
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877)
> > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> > 2010-05-28 07:57:19,901 WARN org.apache.zookeeper.ClientCnxn: Exception
> > closing session 0x128d263e1450001 to sun.nio.ch.SelectionKeyImpl@1878144
> > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> > lim=4 cap=4]
> > at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> > 2010-05-28 07:57:20,244 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> > state: Disconnected, type: None, path: null
> > 2010-05-28 07:57:20,433 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 17 on 60020, call next(3212049130314122049) from
> 192.168.1.115:49741:
> > error: org.apache.hadoop.hbase.LeaseException: lease
> '3212049130314122049'
> > does not exist or has already expired
> > org.apache.hadoop.hbase.LeaseException: lease '3212049130314122049' does
> not
> > exist or has already expired
> > at org.apache.hadoop.hbase.Leases.renewLease(Leases.java:190)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1899)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877)
> > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> > 2010-05-28 07:57:20,539 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 21 on 60020, call next(3787094744445092555) from
> 192.168.1.115:49741:
> > error: org.apache.hadoop.hbase.UnknownScannerException: Name:
> > 3787094744445092555
> > org.apache.hadoop.hbase.UnknownScannerException: Name:
> 3787094744445092555
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1877)
> > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> > 2010-05-28 07:57:21,627 INFO org.apache.zookeeper.ClientCnxn: Attempting
> > connection to server test2/192.168.1.115:2181
> > 2010-05-28 07:57:22,881 INFO org.apache.zookeeper.ClientCnxn: Priming
> > connection to java.nio.channels.SocketChannel[connected local=/
> > 192.168.1.115:37147 remote=test2/192.168.1.115:2181]
> > 2010-05-28 07:57:23,405 INFO org.apache.zookeeper.ClientCnxn: Server
> > connection successful
> > 2010-05-28 07:57:23,664 WARN org.apache.zookeeper.ClientCnxn: Exception
> > closing session 0x128d263e1450001 to sun.nio.ch.SelectionKeyImpl@192343b
> > java.io.IOException: Session Expired
> > at
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
> > at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
> > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> >
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > 2010-05-28 09:29:41,286 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started
> > at: 192.168.1.115:60020
> > 2010-05-28 09:29:41,314 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> > /hbase/root-region-server got 192.168.1.115:60020
> > 2010-05-28 09:29:41,318 INFO
> org.apache.hadoop.hbase.regionserver.StoreFile:
> > Allocating LruBlockCache with maximum size 199.4m
> > 2010-05-28 09:29:41,660 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> > org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2278)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1857)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> >
>