[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files

["Duo Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16633250#comment-16633250 ] 

Duo Zhang commented on HBASE-21254:
-----------------------------------

I think maybe we could introduce something like the force flush at RS side. There is a holdingCleanupTracker for the oldest procedure wal file, if there are still procedure alive and only have record in this file then we can not deleted it, and we can get the procedure id through the holdingCleanupTracker. So maybe we can do this, add a listener call to indicate the upper layer, that these procedures need to be persist again, then the upper layer will call ProcedureStore.update to persist these procedures to the newest procedure wal file, then we are safe to delete the oldest files.

I do not think this will break any assumptions? What do you guys think? [~stack] [~allan163].

> Need to find a way to limit the number of proc wal files
> --------------------------------------------------------
>
>                 Key: HBASE-21254
>                 URL: https://issues.apache.org/jira/browse/HBASE-21254
>             Project: HBase
>          Issue Type: Sub-task
>          Components: proc-v2
>            Reporter: Duo Zhang
>            Priority: Major
>             Fix For: 3.0.0, 2.2.0
>
>
> For regionserver, we have a max wal file limitation, if we reach the limitation, we will trigger a flush on specific regions so that we can delete old wal files. But for proc wals, we do not have this mechanism, and it will be worse after HBASE-21233, as if there is an old procedure which can not make progress and do not persist its state, we need to keep the old proc wal file for ever...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)