[jira] [Updated] (KUDU-1790) "Master lookup permit" design wastes lots of reactor thread CPU

["Todd Lipcon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KUDU-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated KUDU-1790:
------------------------------
    Attachment: fg.svg

Here's a flame graph I captured from a reactor thread. I also noted that 70%+ of the thread's wall time was blocked in futex() syscalls related to this semaphore.

> "Master lookup permit" design wastes lots of reactor thread CPU
> ---------------------------------------------------------------
>
>                 Key: KUDU-1790
>                 URL: https://issues.apache.org/jira/browse/KUDU-1790
>             Project: Kudu
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 1.1.0
>            Reporter: Todd Lipcon
>         Attachments: fg.svg
>
>
> Looking at some stress tests, I see that the reactor thread is spending a significant percent of its CPU time in LookupRpc::SendRpcCb, and doing thousands of futex calls per second. It seems like the current approach of linear backoff of TryWait calls is not a good design, and we should probably do something less busy-wait-like to avoid this problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)