[jira] [Commented] (HBASE-20727) Persist FlushedSequenceId to speed up WAL split after cluster restart

["Bo Cui (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-20727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266480#comment-17266480 ] 

Bo Cui commented on HBASE-20727:
--------------------------------

[~allan163]hi，Can we optimize the feature?
1) the new file is written to the tmp directory. after the new file is written and moved successfully, delete the old file. because new files may fail to be written.
2) can we write new file in batches? If hbase has too many regions, the FlushedSequenceIdFlusher occupies master ChoreService for a long time...like HBASE-25506

> Persist FlushedSequenceId to speed up WAL split after cluster restart
> ---------------------------------------------------------------------
>
>                 Key: HBASE-20727
>                 URL: https://issues.apache.org/jira/browse/HBASE-20727
>             Project: HBase
>          Issue Type: New Feature
>    Affects Versions: 2.0.0
>            Reporter: Allan Yang
>            Assignee: Allan Yang
>            Priority: Major
>             Fix For: 3.0.0-alpha-1
>
>         Attachments: HBASE-20727.002.patch, HBASE-20727.003.patch, HBASE-20727.004.patch, HBASE-20727.005.patch, HBASE-20727.patch
>
>
> We use flushedSequenceIdByRegion and storeFlushedSequenceIdsByRegion in ServerManager to record the latest flushed seqids of regions and stores. So during log split, we can use seqids stored in those maps to filter out the edits which do not need to be replayed. But, those maps are not persisted. After cluster restart or master restart, info of flushed seqids are all lost. 
> Here I offer a way to persist those info to HDFS, even if master restart, we can still use those info to filter WAL edits and then to speed up replay.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)