You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Allan Yang (JIRA)" <ji...@apache.org> on 2018/10/16 15:41:00 UTC

[jira] [Commented] (HBASE-21316) All RegionServer Down when RS_LOG_REPLAY_OPS

    [ https://issues.apache.org/jira/browse/HBASE-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651948#comment-16651948 ] 

Allan Yang commented on HBASE-21316:
------------------------------------

Looks like a corrupted log, can you run 'hbase wal -p walFilePath' to print out the wal, making sure it can finish normally? 

> All RegionServer Down when RS_LOG_REPLAY_OPS
> --------------------------------------------
>
>                 Key: HBASE-21316
>                 URL: https://issues.apache.org/jira/browse/HBASE-21316
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 2.0.0
>            Reporter: justice
>            Priority: Major
>         Attachments: log.tgz
>
>
> 1. One RegionServer die as unknow reason, log as follow:
> {code:java}
> 2018-10-14 20:31:47,423 INFO [main-SendThread(11.3.20.101:2181)] zookeeper.ClientCnxn: Socket connection established to 11.3.20.101/11.3.20.101:2181, initiating session 2018-10-14 20:31:47,433 INFO [main-SendThread(11.3.20.101:2181)] zookeeper.ClientCnxn: Session establishment complete on server 11.3.20.101/11.3.20.101:2181, sessionid = 0x6500073f944a8e79, negotiated timeout = 30000 2018-10-14日 Sunday 21:03:05 CST Starting regionserver on 11-3-19-199.JD.LOCAL core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited
> {code}
> 2. Master receive zk deletenode event, and start ServerCrashProcedure Task
> {code:java}
> 2018-10-14 20:31:47,437 INFO [main-EventThread] master.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [11-3-19-199.jd.local,16020,1539492869470]
> 2018-10-14 20:31:47,539 INFO [PEWorker-1] procedure.ServerCrashProcedure: Start pid=25053, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=11-3-19-199.jd.local,16020,1539492869470, splitWal=true, meta=false
> 2018-10-14 20:31:47,550 INFO [PEWorker-1] master.SplitLogManager: Started splitting 63 logs in [hdfs://11-3-18-67.JD.LOCAL:9000/hbase/WALs/11-3-19-199.jd.local,16020,1539492869470-splitting] for [11-3-19-199.jd.local,16020,1539492869470] ... 2018-10-14 20:31:48,592 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Task /hbase/splitWAL/WALs%2F11-3-19-199.jd.local%2C16020%2C1539492869470-splitting%2F11-3-19-199.jd.local%252C16020%252C1539492869470.1539520250598 acquired by 11-3-18-71.jd.local,16020,1539492869409
> {code}
> 3. One alive RegionServer Node get SplitLogWorker,  has an error and stop
> {code:java}
> 2018-10-14 20:31:48,602 INFO [SplitLogWorker-11-3-18-71:16020] coordination.ZkSplitLogWorkerCoordination: worker 11-3-18-71.jd.local,16020,1539492869409 acquired task /hbase/splitWAL/WALs%2F11-3-19-199.jd.local%2C16020%2C1539492869470-splitting%2F11-3-19-199.jd.local%252C16020%252C1539492869470.1539520250598 
> ...
> 2018-10-14 21:03:26,219 ERROR [RS_LOG_REPLAY_OPS-regionserver/11-3-18-71:16020-1] executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY
> java.lang.ArrayIndexOutOfBoundsException: 8811
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1365)
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1358)
> at org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:735)
> at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816)
> at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:102)
> at org.apache.hadoop.hbase.wal.WALEdit.isMetaEdit(WALEdit.java:107)
> at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:296)
> at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:194)
> at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:99)
> at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-10-14 21:03:26,227 ERROR [RS_LOG_REPLAY_OPS-regionserver/11-3-18-71:16020-1] regionserver.HRegionServer: ***** ABORTING region server 11-3-18-71.jd.local,16020,1539522186368: Caught throwable while processing event RS_LOG_REPLAY *****
> ....
> 2018-10-14 20:31:48,780 INFO [RS_LOG_REPLAY_OPS-regionserver/11-3-18-71:16020-0] regionserver.HRegionServer: ***** STOPPING region server '11-3-18-71.jd.local,16020,1539492869409' *****
> {code}
> 4. other alive regionserver node die one by one, at last, all regionserver node die



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)