You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2008/02/06 23:29:08 UTC
[jira] Commented: (HBASE-12) when hbase regionserver restarts, it
says "impossible state for createLease()"
[ https://issues.apache.org/jira/browse/HBASE-12?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12566379#action_12566379 ]
Jean-Daniel Cryans commented on HBASE-12:
-----------------------------------------
Got pretty much the same thing here. Fetched 0.17-dev, compiled, config hadoop, started dfs, config hbase with just the master adress and regionservers (12 nodes), started hbase and then this. (sorry for long post)
HBase master log :
{code}
...
2008-02-06 16:57:22,131 INFO org.apache.hadoop.hbase.HMaster: Root region dir: /tmp/hadoop-root/hbase/-ROOT-/70236052
2008-02-06 16:57:23,577 INFO org.apache.hadoop.hbase.HMaster: bootstrap: creating ROOT and first META regions
2008-02-06 16:57:23,792 INFO org.apache.hadoop.hbase.HLog: new log writer created at hdfs://192.168.0.1:9000/tmp/hadoop-root/hbase/-ROOT-/70236052/log/hlog.dat.000
2008-02-06 16:57:23,964 INFO org.apache.hadoop.hbase.HRegion: region -ROOT-,,0 available
2008-02-06 16:57:24,014 INFO org.apache.hadoop.hbase.HLog: new log writer created at hdfs://192.168.0.1:9000/tmp/hadoop-root/hbase/.META./1028785192/log/hlog.dat.000
2008-02-06 16:57:24,079 INFO org.apache.hadoop.hbase.HRegion: region .META.,,1 available
2008-02-06 16:57:24,217 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2008-02-06 16:57:25,398 INFO org.apache.hadoop.hbase.HRegion: closed -ROOT-,,0
2008-02-06 16:57:25,474 INFO org.apache.hadoop.hbase.HRegion: closed .META.,,1
2008-02-06 16:57:25,887 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing RPC Metrics with serverName=60000, port=60000
2008-02-06 16:57:26,342 INFO org.apache.hadoop.hbase.HMaster: HMaster initialized on 192.168.0.1:60000
2008-02-06 16:57:26,748 INFO org.mortbay.util.Credential: Checking Resource aliases
2008-02-06 16:57:26,959 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
2008-02-06 16:57:26,963 INFO org.mortbay.util.Container: Started HttpContext[/static,/static]
2008-02-06 16:57:26,963 INFO org.mortbay.util.Container: Started HttpContext[/logs,/logs]
2008-02-06 16:57:28,489 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.WebApplicationHandler@8997d1
2008-02-06 16:57:28,670 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/]
2008-02-06 16:57:29,815 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.WebApplicationHandler@1b7ae22
2008-02-06 16:57:29,827 INFO org.mortbay.util.Container: Started WebApplicationContext[/api,rest]
2008-02-06 16:57:29,839 INFO org.mortbay.http.SocketListener: Started SocketListener on 0.0.0.0:60010
2008-02-06 16:57:29,839 INFO org.mortbay.util.Container: Started org.mortbay.jetty.Server@b301f2
2008-02-06 16:57:29,840 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2008-02-06 16:57:29,842 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 60000: starting
2008-02-06 16:57:29,873 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60000: starting
2008-02-06 16:57:29,874 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60000: starting
2008-02-06 16:57:29,875 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60000: starting
2008-02-06 16:57:29,876 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000: starting
2008-02-06 16:57:29,877 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60000: starting
2008-02-06 16:57:29,878 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60000: starting
2008-02-06 16:57:29,878 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60000: starting
2008-02-06 16:57:29,879 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60000: starting
2008-02-06 16:57:29,880 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60000: starting
2008-02-06 16:57:29,881 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60000: starting
2008-02-06 16:57:29,923 INFO org.apache.hadoop.hbase.HMaster: received start message from: 127.0.0.1:60020
2008-02-06 16:57:29,962 INFO org.apache.hadoop.hbase.HMaster: received start message from: 127.0.0.1:60020
2008-02-06 16:57:29,972 INFO org.apache.hadoop.hbase.HMaster: received start message from: 127.0.0.1:60020
2008-02-06 16:57:29,982 INFO org.apache.hadoop.hbase.HMaster: received start message from: 127.0.0.1:60020
2008-02-06 16:57:29,995 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60000, call regionServerStartup(address: 127.0.0.1:60020, startcode: 1202333840817, load: (requests: 0 regions: 0)) from 192.168.0.5:43524: error: java.io.IOException: java.lang.AssertionError: Impossible state for createLease(): Lease -1109690057/-1109690057 is still held.
java.io.IOException: java.lang.AssertionError: Impossible state for createLease(): Lease -1109690057/-1109690057 is still held.
at org.apache.hadoop.hbase.Leases.createLease(Leases.java:145)
at org.apache.hadoop.hbase.HMaster.regionServerStartup(HMaster.java:1310)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
... many times
and finally
2008-02-06 17:14:44,701 INFO org.apache.hadoop.hbase.HMaster: Cancelling lease for 127.0.0.1:60020
2008-02-06 17:14:44,701 INFO org.apache.hadoop.hbase.HMaster: received start message from: 127.0.0.1:60020
2008-02-06 17:14:44,701 INFO org.apache.hadoop.hbase.HMaster: Region server 127.0.0.1:60020: MSG_REPORT_EXITING -- lease cancelled
2008-02-06 17:14:45,289 INFO org.apache.hadoop.hbase.HMaster: Cancelling lease for 127.0.0.1:60020
2008-02-06 17:14:46,901 INFO org.apache.hadoop.hbase.HMaster: All user tables quiesced. Proceeding with shutdown
2008-02-06 17:14:46,904 INFO org.apache.hadoop.hbase.HMaster$MetaScanner: HMaster.metaScanner exiting
2008-02-06 17:14:47,161 INFO org.apache.hadoop.hbase.HMaster: All user tables quiesced. Proceeding with shutdown
2008-02-06 17:14:47,587 INFO org.apache.hadoop.hbase.HMaster: All user tables quiesced. Proceeding with shutdown
2008-02-06 17:14:47,628 INFO org.apache.hadoop.hbase.HMaster: All user tables quiesced. Proceeding with shutdown
2008-02-06 17:14:48,936 INFO org.apache.hadoop.hbase.HMaster: Stopping infoServer
2008-02-06 17:14:48,937 INFO org.mortbay.util.ThreadedServer: Stopping Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60010]
2008-02-06 17:14:48,967 INFO org.mortbay.http.SocketListener: Stopped SocketListener on 0.0.0.0:60010
2008-02-06 17:14:49,307 INFO org.mortbay.util.Container: Stopped HttpContext[/static,/static]
2008-02-06 17:14:49,756 INFO org.mortbay.util.Container: Stopped HttpContext[/logs,/logs]
2008-02-06 17:14:49,757 INFO org.mortbay.util.Container: Stopped org.mortbay.jetty.servlet.WebApplicationHandler@1a4c5b4
2008-02-06 17:14:50,087 INFO org.mortbay.util.Container: Stopped WebApplicationContext[/,/]
2008-02-06 17:14:50,087 INFO org.mortbay.util.Container: Stopped org.mortbay.jetty.servlet.WebApplicationHandler@5b675e
2008-02-06 17:14:50,466 INFO org.mortbay.util.Container: Stopped WebApplicationContext[/api,rest]
2008-02-06 17:14:50,467 INFO org.mortbay.util.Container: Stopped org.mortbay.jetty.Server@a00185
2008-02-06 17:14:50,467 INFO org.apache.hadoop.ipc.Server: Stopping server on 60000
2008-02-06 17:14:50,469 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60000: exiting
2008-02-06 17:14:50,469 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60000: exiting
2008-02-06 17:14:50,469 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60000: exiting
2008-02-06 17:14:50,470 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60000: exiting
2008-02-06 17:14:50,470 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60000: exiting
2008-02-06 17:14:50,470 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60000: exiting
2008-02-06 17:14:50,470 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60000: exiting
2008-02-06 17:14:50,471 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 60000
2008-02-06 17:14:50,471 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000: exiting
2008-02-06 17:14:50,472 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60000: exiting
2008-02-06 17:14:50,473 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60000: exiting
2008-02-06 17:14:50,483 INFO org.apache.hadoop.hbase.Leases: HMaster closing leases
2008-02-06 17:14:50,484 INFO org.apache.hadoop.hbase.Leases$LeaseMonitor: HMaster.leaseChecker exiting
2008-02-06 17:14:50,485 INFO org.apache.hadoop.hbase.Leases: HMaster closed leases
2008-02-06 17:14:50,495 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
{code}
regionservers stop after few seconds, unable to stop master properly.
> when hbase regionserver restarts, it says "impossible state for createLease()"
> ------------------------------------------------------------------------------
>
> Key: HBASE-12
> URL: https://issues.apache.org/jira/browse/HBASE-12
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Michael Bieniosek
> Assignee: Jim Kellerman
> Priority: Minor
> Fix For: 0.2.0
>
>
> I restarted a regionserver, and got this error in its logs:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.AssertionError: Impossible state for createLease(): Lease -435227488/-435227488 is still held.
> at org.apache.hadoop.hbase.Leases.createLease(Leases.java:145)
> at org.apache.hadoop.hbase.HMaster.regionServerStartup(HMaster.java:1278
> )
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> at org.apache.hadoop.ipc.Client.call(Client.java:482)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
> at $Proxy0.regionServerStartup(Unknown Source)
> at org.apache.hadoop.hbase.HRegionServer.reportForDuty(HRegionServer.jav
> a:1025)
> at org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:659)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.