[jira] [Commented] (KUDU-1587) Memory-based backpressure is insufficient on seek-bound workloads

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15448127#comment-15448127 ] 

Todd Lipcon commented on KUDU-1587:
-----------------------------------

The issue here is that the only backpressure on new operations being submitted to the apply queue is based on the TransactionTracker memory limits. Each operation here is only about 1KB of inserted data (or less) and it results in a half second of worker wall time. With the default 64MB tracker limit (per tablet), that's enough space for 64000x500ms = 32000 seconds worth of work. Assume 10x parallelism due to many disks, but then recall that the limiting is per-tablet and we have 10+ active tablets, which gets us back into the 30,000-40,000 seconds worth of available queueing before backpressure kicks in.

> Memory-based backpressure is insufficient on seek-bound workloads
> -----------------------------------------------------------------
>
>                 Key: KUDU-1587
>                 URL: https://issues.apache.org/jira/browse/KUDU-1587
>             Project: Kudu
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 0.10.0
>            Reporter: Todd Lipcon
>            Priority: Critical
>         Attachments: queue-time.png
>
>
> I pushed a uniform random insert workload from a bunch of clients to the point that the vast majority of bloom filters no longer fit in buffer cache, and the compaction had fallen way behind. Thus, every inserted row turns into 40+ seeks (due to non-compact data) and takes 400-500ms. In this kind of workload, the current backpressure (based on memory usage) is insufficient to prevent ridiculously long queues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)