[jira] [Commented] (KUDU-1622) ResultCache exhibits high contention under non-batched write workload

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711152#comment-15711152 ] 

Todd Lipcon commented on KUDU-1622:
-----------------------------------

In an Impala stress test I"m also seeing lots of blocked threads in ResultTracker code.

It seems like in almost all cases the thread holding the lock is in this code path:

{code}
    if (MustHandleRpc(handler_attempt_no, completion_record, ongoing_rpc)) {
      if (ongoing_rpc.context != nullptr) {
        func(ongoing_rpc);
        delete ongoing_rpc.context;
      }
{code}

Would it be possible to just accumulate a list of ongoing_rpcs to respond to, then drop the lock before calling func() on each and deleting the contexts?

> ResultCache exhibits high contention under non-batched write workload
> ---------------------------------------------------------------------
>
>                 Key: KUDU-1622
>                 URL: https://issues.apache.org/jira/browse/KUDU-1622
>             Project: Kudu
>          Issue Type: Improvement
>          Components: perf, rpc
>    Affects Versions: 1.0.0
>            Reporter: Todd Lipcon
>         Attachments: fg.svg, pprof13384.0.svg
>
>
> I am running a YCSB workload with non-batched writes, and I see ResultCache among the highest CPU consumers (>15% of the CPU). There's a lot of lock contention on the result cache map. We should consider a more concurrent data structure and/or finer-grained locking.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)