You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org> on 2012/05/09 17:21:50 UTC
[jira] [Commented] (HBASE-5972) RegionServer Hangs on Compaction
upon NameNode Death
[ https://issues.apache.org/jira/browse/HBASE-5972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271479#comment-13271479 ]
Nicolas Spiegelberg commented on HBASE-5972:
--------------------------------------------
RS lost its znode at around 9:02. .. but kept dragging on until 9:22 am to finally die.
2012-05-01 09:02:47,617 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 32 on 60020: exiting
2012-05-01 09:02:47,616 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 289 on 60020: exiting
2012-05-01 09:02:47,617 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 51 on 60020: exiting
2012-05-01 09:03:38,395 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2012-05-01 09:04:38,395 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2012-05-01 09:05:38,396 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:06:38,396 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:07:38,396 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2012-05-01 09:08:38,396 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2012-05-01 09:09:38,396 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:10:38,422 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:11:38,472 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:12:38,472 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:13:38,473 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:14:38,473 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:15:38,482 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:16:38,482 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:17:38,482 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:18:38,483 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:19:38,483 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:20:38,483 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:21:38,483 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...2012-05-01 09:22:38,483 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2012-05-01 09:22:40,378 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /TITAN019-ASH3-HBASE/MailBox_cell19_v1/a
296dd72364afee146190f5a751606eb/.tmp/ca501bfd82474bd39a94f5aff0e08709 File does not exist. [Lease. Holder: DFSClient_-1458957666, pendingcreates: 1] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1830)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1821) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1665) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:600)
at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:737)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1065)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1061)
at java.security.AccessController.doPrivileged(Native Method)
> RegionServer Hangs on Compaction upon NameNode Death
> ----------------------------------------------------
>
> Key: HBASE-5972
> URL: https://issues.apache.org/jira/browse/HBASE-5972
> Project: HBase
> Issue Type: Bug
> Reporter: Nicolas Spiegelberg
>
> Noticed 20-minute hangs when doing a namenode kill test on our production shadow cluster. All regionservers terminated except one, which got stuck with the following. There might be some race conditions/deadlocks on compaction thread shutdown worth investigating.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira