You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@hbase.apache.org by "Prathyusha (Jira)" <ji...@apache.org> on 2026/10/22 19:53:00 UTC

[jira] [Created] (HBASE-30493) SplitWALProcedure creates SplitWALRemoteProcedures in a loop when WAL archival fails without any retry strategy

Prathyusha created HBASE-30493:
----------------------------------

             Summary: SplitWALProcedure creates SplitWALRemoteProcedures in a loop when WAL archival fails without any retry strategy
                 Key: HBASE-30493
                 URL: https://issues.apache.org/jira/browse/HBASE-30493
             Project: HBase
          Issue Type: Bug
            Reporter: Prathyusha


SplitWALProcedure creates millions of SplitWALRemoteProcedures in a loop when WAL archival fails without any retry strategy.
On our new cluster due to a directory file limit issue at HDFS, WAL archival failed and we saw 
~11 Million SplitWALRemoteProcedures created in a loop
Storming the Procedure store
blocked all SCPs, bringing the cluster down 
MasterData/WALs reached around ~22.5K, which on crash resulted in 6-7 hours of startup time
HMaster memory had to be increased to accommodate the loading of procedure store at startup (this is cause all the SplitWALRemoteProcedures are unfinished childs of SCPs).

Proposed fix
Instead of creating SplitWALRemoteProcedures in a loop, we attach a retry strategy [here |https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitWALProcedure.java#L96]



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)