You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "mazhengxuan (Jira)" <ji...@apache.org> on 2026/10/20 13:00:00 UTC
[jira] [Commented] (HBASE-28932) Abort RS if unable to sync internal markers.
[ https://issues.apache.org/jira/browse/HBASE-28932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129132#comment-18129132 ]
mazhengxuan commented on HBASE-28932:
-------------------------------------
I looked into the current code. ReplicationMarkerChore only logs the sync failure, and compaction failures can also return without aborting the RS.
Other paths, such as bulk-load markers and flush commit markers, already have abort handling.
WALUtil explicitly leaves the abort decision to callers, following HBASE-27230. Would it make sense to handle the final WAL sync timeout in AbstractFSWAL.blockOnSync using its existing Abortable? This would cover FSHLog and AsyncFSWAL, including standalone sync calls outside WALUtil.
I suggest limiting the initial change to actual WAL sync timeouts and preserving the existing handling of other exceptions. Is that the intended scope, or should non-timeout sync failures also be fatal?
> Abort RS if unable to sync internal markers.
> --------------------------------------------
>
> Key: HBASE-28932
> URL: https://issues.apache.org/jira/browse/HBASE-28932
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 2.5.8
> Reporter: Rushabh Shah
> Priority: Major
>
> RS kept on running even if it was unable to write replication marker for 5 minutes. But this issue is not specific to just replication marker. It applies to compaction marker as well as region event marker (like open, close).
> Sample exception trace:
> {noformat}
> 2024-10-09 10:12:21,659 ERROR [regionserver/regionserver-33:60020.Chore.3] regionserver.ReplicationMarkerChore - Exception whil
> e sync'ing replication tracker edit
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=15030132, WAL system stuck?
> at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:171)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:876)
> at org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:802)
> at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doSync(FSHLog.java:836)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$sync$3(AbstractFSWAL.java:602)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.sync(AbstractFSWAL.java:602)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.sync(AbstractFSWAL.java:592)
> at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullMarkerAppendTransaction(WALUtil.java:169)
> at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:146)
> at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeReplicationMarkerAndSync(WALUtil.java:230)
> at org.apache.hadoop.hbase.replication.regionserver.ReplicationMarkerChore.chore(ReplicationMarkerChore.java:99)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:161)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> {noformat}
> In this case there was namenode crash and all the hdfs clients on regionservers didn't failover to new active namenode due to our inefficient failover configuration parameters.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)