[jira] [Commented] (KUDU-1954) Improve maintenance manager behavior in heavy write workload

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936844#comment-15936844 ] 

Todd Lipcon commented on KUDU-1954:
-----------------------------------

Started writing various notes here:
https://docs.google.com/document/d/17-2CcmrjxZY0Gd9wDUh83xCCNL574famw8_2Bhfu-_8/edit?usp=sharing

> Improve maintenance manager behavior in heavy write workload
> ------------------------------------------------------------
>
>                 Key: KUDU-1954
>                 URL: https://issues.apache.org/jira/browse/KUDU-1954
>             Project: Kudu
>          Issue Type: Improvement
>          Components: perf, tserver
>    Affects Versions: 1.3.0
>            Reporter: Todd Lipcon
>         Attachments: mm-trace.png
>
>
> During the investigation in [this doc|https://docs.google.com/document/d/1U1IXS1XD2erZyq8_qG81A1gZaCeHcq2i0unea_eEf5c/edit] I found a few maintenance-manager-related issues during heavy writes:
> - we don't schedule flushes until we are already in "backpressure" realm, so we spent most of our time doing backpressure
> - even if we configure N maintenance threads, we typically are only using ~50% of those threads due to the scheduling granularity
> - when we do hit the "memory-pressure flush" threshold, all threads quickly switch to flushing, which then brings us far beneath the threshold
> - long running compactions can temporarily starve flushes
> - high volume of writes can starve compactions



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)