You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Xiaolin Ha (Jira)" <ji...@apache.org> on 2022/03/09 13:56:00 UTC
[jira] [Comment Edited] (HBASE-22761) Caught ArrayIndexOutOfBoundsException while processing event RS_LOG_REPLAY
[ https://issues.apache.org/jira/browse/HBASE-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503599#comment-17503599 ]
Xiaolin Ha edited comment on HBASE-22761 at 3/9/22, 1:55 PM:
-------------------------------------------------------------
Though we can make wal entries not be loss, but we can not ensure that every wal entry be flushed exactly once, right? Then if an entry is flushed twice, the AsyncFSWAL#unackedAppends will release the entry twice, the second flushed entry may have dirty data by the released direct bytebuffer.
Here is a sceniano,
# AsyncFSWAL#toWriteAppends=1,2,3,4,5,6,7,8,9,10...
# sync entries 1,2,3, whose total size is up to the configured batch size. toWriteAppends=4,5,6,7,8,9, unackedAppends=1,2,3;
# sync entries 4,5,6, toWriteAppends=7,8,9, unackedAppends=1,2,3,4,5,6;
# sync entries 7,8,9, toWriteAppends=10..., unackedAppends=1,2,3,4,5,6,7,8,9;
# 1,2,3 sync failed, then all the entries in the unackedAppends are added back to toWriteAppends, toWriteAppends=1,2,3,4,5,6,7,8,9,10...., unackedAppends=1,2,3,4,5,6,7,8,9;
# wal rolled success, just caused by writer broken;
# the last epoch wal writer flushed 4,5,6 fail, but nothing happens now, the toWriteAppends=1,2,3,4,5,6,7,8,9,10...., unackedAppends=1,2,3,4,5,6,7,8,9;
# the last epoch wal writer flushed 7,8,9 success, then the unackedAppends=[], 1,2,3,4,5,6,7,8,9 entries are released (but they are in the toWriteAppends and waiting for the next new writer to flush);
# the new epoch wal writer sync entries 1,2,3, toWriteAppends=4,5,6,7,8,9, unackedAppends=1,2,3;
# 1,2,3 are flushed success, but the direct bytebuffer they refered to has already be released and may have been written new data.
[~zhangduo] [~comnetwork] what do you think?
was (Author: xiaolin ha):
Though we can make wal entries not be loss, but we can not ensure that every wal entry be flushed exactly once, right? Then if an entry is flushed twice, the AsyncFSWAL#unackedAppends will release the entry twice, the second flushed entry may have dirty data by the released direct bytebuffer.
Here is a sceniano,
# AsyncFSWAL#toWriteAppends=1,2,3,4,5,6,7,8,9,10...
# sync entries 1,2,3, whose total size is up to the configured batch size. toWriteAppends=4,5,6,7,8,9, unackedAppends=1,2,3;
# sync entries 4,5,6, toWriteAppends=7,8,9, unackedAppends=1,2,3,4,5,6;
# sync entries 7,8,9, toWriteAppends=10..., unackedAppends=1,2,3,4,5,6,7,8,9;
# 1,2,3 sync failed, then all the entries in the unackedAppends are added back to toWriteAppends, toWriteAppends=1,2,3,4,5,6,7,8,9,10...., unackedAppends=1,2,3,4,5,6,7,8,9;
# wal rolled success, just caused by writer broken;
# the last epoch wal writer flushed 4,5,6 fail, but nothing happens now, the toWriteAppends=1,2,3,4,5,6,7,8,9,10...., unackedAppends=1,2,3,4,5,6,7,8,9;
# the last epoch wal writer flushed 7,8,9 success, then the unackedAppends=[], 1,2,3,4,5,6,7,8,9 entries are released (but they are in the toWriteAppends and waiting for the next new writer to flush);
# the new epotch wal writer sync entries 1,2,3, toWriteAppends=4,5,6,7,8,9, unackedAppends=1,2,3;
# 1,2,3 are flushed success, but the direct bytebuffer they refered to has already be released and may have been written new data.
[~zhangduo] [~comnetwork] what do you think?
> Caught ArrayIndexOutOfBoundsException while processing event RS_LOG_REPLAY
> --------------------------------------------------------------------------
>
> Key: HBASE-22761
> URL: https://issues.apache.org/jira/browse/HBASE-22761
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.1.1
> Reporter: casuallc
> Priority: Major
> Attachments: tmp
>
>
> RegionServer exists when error happen
> {code:java}
> 2019-07-29 20:51:09,726 INFO [RS_LOG_REPLAY_OPS-regionserver/h1:16020-0] wal.WALSplitter: Processed 0 edits across 0 regions; edits skipped=0; log file=hdfs://cluster1/hbase/WALs/h2,16020,1564216856546-splitting/h2%2C16020%2C1564216856546.1564398538121, length=615233, corrupted=false, progress failed=false
> 2019-07-29 20:51:09,726 INFO [RS_LOG_REPLAY_OPS-regionserver/h1:16020-0] handler.WALSplitterHandler: Worker h1,16020,1564404572589 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@577da0d3 in 84892ms. Status = null
> 2019-07-29 20:51:09,726 ERROR [RS_LOG_REPLAY_OPS-regionserver/h1:16020-0] executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY
> java.lang.ArrayIndexOutOfBoundsException: 16403
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1365)
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1358)
> at org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:735)
> at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816)
> at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:143)
> at org.apache.hadoop.hbase.wal.WALEdit.isMetaEdit(WALEdit.java:148)
> at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297)
> at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:195)
> at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:100)
> at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2019-07-29 20:51:09,730 ERROR [RS_LOG_REPLAY_OPS-regionserver/h1:16020-0] regionserver.HRegionServer: ***** ABORTING region server h1,16020,1564404572589: Caught throwable while processing event RS_LOG_REPLAY *****
> java.lang.ArrayIndexOutOfBoundsException: 16403
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1365)
> at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1358)
> at org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:735)
> at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816)
> at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:143)
> at org.apache.hadoop.hbase.wal.WALEdit.isMetaEdit(WALEdit.java:148)
> at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297)
> at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:195)
> at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:100)
> at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)