[jira] [Commented] (KUDU-1408) Adding a replica may never succeed if copying tablet takes longer than the log retention time

["Jean-Daniel Cryans (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365047#comment-15365047 ] 

Jean-Daniel Cryans commented on KUDU-1408:
------------------------------------------

KUDU-1512 has logs that show this in action.

What really worries me about this is that with replication=3 there's no room for machine failures (like in KUDU-1512). Start a big insert, have a few slow nodes, one node dies, suddenly you can't use a bunch of tablets. Being next to the PONR should be considered a crisis. It would be different with replication=5, because one more node can fail until you're at the PONR.

> Adding a replica may never succeed if copying tablet takes longer than the log retention time
> ---------------------------------------------------------------------------------------------
>
>                 Key: KUDU-1408
>                 URL: https://issues.apache.org/jira/browse/KUDU-1408
>             Project: Kudu
>          Issue Type: Bug
>          Components: consensus, tserver
>    Affects Versions: 0.8.0
>            Reporter: Todd Lipcon
>            Priority: Critical
>
> Currently, while a remote bootstrap session is in progress, we anchor the logs from the time at which it started. However, as soon as the session finishes, we drop the anchor, and delete any logs. In the case where the tablet copy itself takes longer than the log retention period, this means it's likely to have a scenario like:
> - TS A starts downloading from TS B. It plans to download segments 1-4 and adds an anchor.
> - TS B handles writes for 20 minutes, rolling the log many times (e.g. up to log segment 20)
> - TS A finishes downloading, and ends the remote bootstrap session
> - TS B no longer has an anchor, so GCs all logs 1-16.
> - TS A finishes opening the tablet it just copied, but immediately is unable to catch up (because it only has segments 1-4, but the leader only has 17-20)
> - TS B evicts TS A
> This loop will go on basically forever until the write workload stops on TS B.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)