[jira] [Commented] (KUDU-1964) OpenSSL locks create scalability bottleneck

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954645#comment-15954645 ] 

Todd Lipcon commented on KUDU-1964:
-----------------------------------

I think most of the contention is caused by various global mutexes called by ERR_clear_error(). So, I guess one option is that we could try to remove all of our defensive ERR_clear_error()  calls before making SSL API calls and instead make sure to ERR_clear_error *after* any call which might push an error. To make it less error prone we could keep a DCHECK_EQ(ERR_peek_error(), 0) in all the places we currently call clear_error.

> OpenSSL locks create scalability bottleneck
> -------------------------------------------
>
>                 Key: KUDU-1964
>                 URL: https://issues.apache.org/jira/browse/KUDU-1964
>             Project: Kudu
>          Issue Type: Bug
>          Components: impala, perf, security
>    Affects Versions: 1.3.0
>            Reporter: Todd Lipcon
>
> I'm running a concurrent workload against impala and took a pstack where I noticed that most of the kudu threads are stuck in a stack that looks like:
> #6  0x000000348d6e807b in ERR_get_state () from /usr/lib64/libcrypto.so.10
> #7  0x000000348d6e881d in ERR_clear_error () from /usr/lib64/libcrypto.so.10
> #8  0x00007f33fdecb834 in kudu::security::TlsHandshake::Continue(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*) () from /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.16/lib/impala/lib/libkudu_client.so.0
> https://konradzemek.com/2015/08/16/asio-ssl-and-scalability/ notes the same issue but unfortnuately has no good solution aside from switching to BoringSSL



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)