[jira] [Updated] (KUDU-607) Flush MRS/DMS based on how much logs they retain

["Mike Percy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KUDU-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mike Percy updated KUDU-607:
----------------------------
    Parent: KUDU-518

> Flush MRS/DMS based on how much logs they retain
> ------------------------------------------------
>
>                 Key: KUDU-607
>                 URL: https://issues.apache.org/jira/browse/KUDU-607
>             Project: Kudu
>          Issue Type: Sub-task
>          Components: tserver
>    Affects Versions: M5
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>
> The time it takes to bootstrap a tablet after a TS restart is directly dependant on amount of data we have in the logs. We currently have a process that GCs the logs whose data is fully flushed, up to a configurable retention level, meaning that by default we keep at least 2 log segments even if their data is flushed. The reason we want this is so that we some logs around when a follower slows down and we have to help them catch up (instead of kicking them out).
> This starts working against us when some DMS or the MRS only receives a trickle of inserts. We can add time-based flushing but it's not perfect. Instead, we can flush based on whoevers preventing us to GC the biggest amount of logs data. We can do this by using the oldest entry index from the MRS/DMS and check that against the max indexes in each log segments that we can't GC. We then modify the Maintenance Manager to be able to take that into account when selecting an operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)