You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Andrew Wong (Jira)" <ji...@apache.org> on 2020/06/02 21:20:00 UTC
[jira] [Commented] (KUDU-2050) Avoid peer eviction during block
manager startup
[ https://issues.apache.org/jira/browse/KUDU-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17124327#comment-17124327 ]
Andrew Wong commented on KUDU-2050:
-----------------------------------
As a workaround for now, folks can use {{kudu tserver state enter_maintenance}} to avoid re-replication at startup.
> Avoid peer eviction during block manager startup
> ------------------------------------------------
>
> Key: KUDU-2050
> URL: https://issues.apache.org/jira/browse/KUDU-2050
> Project: Kudu
> Issue Type: Bug
> Components: fs, tserver
> Affects Versions: 1.4.0
> Reporter: Adar Dembo
> Priority: Critical
> Labels: stability, supportability
>
> In larger deployments we've observed that opening the block manager can take a really long time, like tens of minutes or sometimes even hours. This is especially true as of 1.4 where the log block manager tries to optimize on-disk data structures during startup.
> The default time to Raft peer eviction is 5 minutes. If one node is restarted and LBM startup takes over 5 minutes, or if all nodes are restarted and there's over 5 minutes of LBM startup time variance across them, the "slow" node could have all of its replicas evicted. Besides generating a lot of unnecessary work in rereplication, this effectively "defeats" the LBM optimizations in that it would have been equally slow (but more efficient) to reformat the node instead.
> So, let's reorder startup such that LBM startup counts towards replica bootstrapping. One idea: adjust FsManager startup so that tablet-meta/cmeta files can be accessed early to construct bootstrapping replicas, but to defer opening of the block manager until after that time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)