You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Nick Dimiduk (Jira)" <ji...@apache.org> on 2024/09/16 15:00:00 UTC
[jira] [Commented] (HBASE-28803) HBase Master stuck due to improper handling of WALSyncTimeoutException within UncheckedIOException
[ https://issues.apache.org/jira/browse/HBASE-28803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882093#comment-17882093 ]
Nick Dimiduk commented on HBASE-28803:
--------------------------------------
What are you thinking here [~psomogyi]? Should there be special handling for this new WALSyncTimeoutIOException in the ProcedureExecutor harness? I'm tracing through the code and it looks to me like all exceptions thrown during procedure execution are WARN logged and swallowed. We can catch this specific exception and thread through a server abort, but maybe we want something more general than that?
> HBase Master stuck due to improper handling of WALSyncTimeoutException within UncheckedIOException
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-28803
> URL: https://issues.apache.org/jira/browse/HBASE-28803
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 2.6.0, 3.0.0-alpha-4
> Reporter: Peter Somogyi
> Assignee: Peter Somogyi
> Priority: Critical
>
> One of our test clusters stuck during a rolling restart due to a WAL.sync timeout. This issue did not result in the Master aborting because the WALSyncTimeoutException was wrapped in an UncheckedIOException, which prevented the proper exception handling mechanism from being triggered. As a result, the Master was handing for a long time and procedures were stuck.
> This was a 2.4 based HBase with HBASE-27230.
> {noformat}
> 2024-08-17 17:23:07,567 ERROR org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore: Failed to delete pid=2027
> org.apache.hadoop.hbase.regionserver.wal.WALSyncTimeoutIOException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=4347, WAL system stuck?
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:848)
> at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:718)
> at org.apache.hadoop.hbase.regionserver.HRegion.sync(HRegion.java:8902)
> at org.apache.hadoop.hbase.regionserver.HRegion.doWALAppend(HRegion.java:8469)
> at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4523)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4447)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4377)
> at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4853)
> at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4847)
> at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4843)
> at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155)
> at org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.lambda$delete$8(RegionProcedureStore.java:379)
> at org.apache.hadoop.hbase.master.region.MasterRegion.update(MasterRegion.java:141)
> at org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.delete(RegionProcedureStore.java:379)
> at org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.delete(RegionProcedureStore.java:410)
> at org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner.periodicExecute(CompletedProcedureCleaner.java:135)
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeInMemoryChore(TimeoutExecutorThread.java:122)
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:101)
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68)
> Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=4347, WAL system stuck?
> at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:171)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:844)
> ... 18 more
> 2024-08-17 17:23:07,568 ERROR org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread: Ignoring pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner exception: org.apache.hadoop.hbase.regionserver.wal.WALSyncTimeoutIOException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=4347, WAL system stuck?
> java.io.UncheckedIOException: org.apache.hadoop.hbase.regionserver.wal.WALSyncTimeoutIOException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=4347, WAL system stuck?
> at org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.delete(RegionProcedureStore.java:383)
> at org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.delete(RegionProcedureStore.java:410)
> at org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner.periodicExecute(CompletedProcedureCleaner.java:135)
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeInMemoryChore(TimeoutExecutorThread.java:122)
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:101)
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68)
> Caused by: org.apache.hadoop.hbase.regionserver.wal.WALSyncTimeoutIOException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=4347, WAL system stuck?
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:848)
> at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:718)
> at org.apache.hadoop.hbase.regionserver.HRegion.sync(HRegion.java:8902)
> at org.apache.hadoop.hbase.regionserver.HRegion.doWALAppend(HRegion.java:8469)
> at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4523)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4447)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4377)
> at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4853)
> at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4847)
> at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4843)
> at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155)
> at org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.lambda$delete$8(RegionProcedureStore.java:379)
> at org.apache.hadoop.hbase.master.region.MasterRegion.update(MasterRegion.java:141)
> at org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.delete(RegionProcedureStore.java:379)
> ... 5 more
> Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=4347, WAL system stuck?
> at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:171)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:844)
> ... 18 more
> 2024-08-17 17:23:07,569 WARN org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK Region-In-Transition state=OPEN, location=host-10.example.com,22101,1723906425777, table=OMID_COMMIT_TABLE, region=1b8c62897ed9e90955e299bfca1e7aa9{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)