You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "mazhengxuan (Jira)" <ji...@apache.org> on 2026/10/20 13:00:00 UTC

[jira] [Commented] (HBASE-28932) Abort RS if unable to sync internal markers.

    [ https://issues.apache.org/jira/browse/HBASE-28932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129132#comment-18129132 ] 

mazhengxuan commented on HBASE-28932:
-------------------------------------

I looked into the current code. ReplicationMarkerChore only logs the sync failure, and compaction failures can also return without aborting the RS.

Other paths, such as bulk-load markers and flush commit markers, already have abort handling.

WALUtil explicitly leaves the abort decision to callers, following HBASE-27230. Would it make sense to handle the final WAL sync timeout in AbstractFSWAL.blockOnSync using its existing Abortable? This would cover FSHLog and AsyncFSWAL, including standalone sync calls outside WALUtil.

I suggest limiting the initial change to actual WAL sync timeouts and preserving the existing handling of other exceptions. Is that the intended scope, or should non-timeout sync failures also be fatal?

> Abort RS if unable to sync internal markers.
> --------------------------------------------
>
>                 Key: HBASE-28932
>                 URL: https://issues.apache.org/jira/browse/HBASE-28932
>             Project: HBase
>          Issue Type: Bug
>          Components: wal
>    Affects Versions: 2.5.8
>            Reporter: Rushabh Shah
>            Priority: Major
>
> RS kept on running even if it was unable to write replication marker for 5 minutes. But this issue is not specific to just replication marker. It applies to compaction marker as well as region event marker (like open, close).
> Sample exception trace:
> {noformat}
> 2024-10-09 10:12:21,659 ERROR [regionserver/regionserver-33:60020.Chore.3] regionserver.ReplicationMarkerChore - Exception whil
> e sync'ing replication tracker edit
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=15030132, WAL system stuck?
>         at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:171)
>         at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:876)
>         at org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:802)
>         at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doSync(FSHLog.java:836)
>         at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$sync$3(AbstractFSWAL.java:602)
>         at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
>         at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.sync(AbstractFSWAL.java:602)
>         at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.sync(AbstractFSWAL.java:592)
>         at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullMarkerAppendTransaction(WALUtil.java:169)
>         at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:146)
>         at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeReplicationMarkerAndSync(WALUtil.java:230)
>         at org.apache.hadoop.hbase.replication.regionserver.ReplicationMarkerChore.chore(ReplicationMarkerChore.java:99)
>         at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:161)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>         at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:750)
> {noformat}
> In this case there was namenode crash and all the hdfs clients on regionservers didn't failover to new active namenode due to our inefficient failover configuration parameters.
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)