[jira] [Resolved] (HBASE-18971) Limit the concurrent opened wal writers when splitting

["Duo Zhang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Duo Zhang resolved HBASE-18971.
-------------------------------
    Resolution: Duplicate

Duplicated by HBASE-19358.

> Limit the concurrent opened wal writers when splitting
> ------------------------------------------------------
>
>                 Key: HBASE-18971
>                 URL: https://issues.apache.org/jira/browse/HBASE-18971
>             Project: HBase
>          Issue Type: Improvement
>          Components: Recovery, wal
>            Reporter: Duo Zhang
>
> A whole cluster restart is very easy to fail under the current architecture if there are many regions on a single region server.
> On a small cluster, although an recovered edits file is very small, NN will reserve a block size for it when opening, so it will easily run out of space.
> And on a large cluster, although the max xceiver count is already 4096, it is still easy to run out of quota and cause DN to reject our request if there are 1k+ regions on a single RS as we will write 3 copies for a block.
> Under the current architecture we need to carefully choose the ‘hbase.regionserver.wal.max.splitters’ and 'hbase.master.executor.serverops.threads' to limit the concurrency of wal splitter. But this is only a compromise as it also slows down the fail recovery.
> So here we want to limit the concurrent opened wal writers when splitting. It may work like a memstore, which buffers the wal entries in memory and when it is full we flush some entries out.
> Suggestions are welcomed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)