You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Adar Dembo (JIRA)" <ji...@apache.org> on 2019/02/12 23:37:00 UTC

[jira] [Created] (KUDU-2700) Memory leak in kudu-tool-test when using OpenSSL 1.1

Adar Dembo created KUDU-2700:
--------------------------------

             Summary: Memory leak in kudu-tool-test when using OpenSSL 1.1
                 Key: KUDU-2700
                 URL: https://issues.apache.org/jira/browse/KUDU-2700
             Project: Kudu
          Issue Type: Bug
          Components: rpc
    Affects Versions: 1.8.0
            Reporter: Adar Dembo


KUDU-1889 added support for OpenSSL 1.1, but one of the outstanding issues is a memory leak reported by LSAN when running tests that use the CLI to connect to a remote server. The memory leak looks like this:
{noformat}
=================================================================
==31426==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 584 byte(s) in 1 object(s) allocated from:
    #0 0x620470 in __interceptor_malloc /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/li
b/asan/asan_malloc_linux.cc:88
    #1 0x7fa17eba94b8 in CRYPTO_zalloc (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x1574b8)
    #2 0x7fa17eb8afdd in ERR_get_state (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x138fdd)
    #3 0x7fa17eb8b213  (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x139213)
    #4 0x7fa17f44e28d in kudu::security::internal::ScopedCheckNoPendingSSLErrors::ScopedCheckNoPendingSSLErrors(char
 const*) ../src/kudu/security/openssl_util.h:201:5
    #5 0x7fa17f49fd51 in kudu::security::TlsSocket::Writev(iovec const*, int, long*) ../src/kudu/security/tls_socket
.cc:81:3
    #6 0x7fa18010321a in kudu::rpc::OutboundTransfer::SendBuffer(kudu::Socket&) ../src/kudu/rpc/transfer.cc:218:26
    #7 0x7fa17fff8eb6 in kudu::rpc::Connection::WriteHandler(ev::io&, int) ../src/kudu/rpc/connection.cc:791:31
    #8 0x7fa17bbaf698 in ev_invoke_pending /home/adar/Source/kudu/thirdparty/src/libev-4.20/ev.c:3155
    #9 0x7fa1800616c9 in kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) ../src/kudu/rpc/reactor.cc:176:3
    #10 0x7fa17bbb05a0 in ev_run /home/adar/Source/kudu/thirdparty/src/libev-4.20/ev.c:3555
    #11 0x7fa180061a18 in kudu::rpc::ReactorThread::RunThread() ../src/kudu/rpc/reactor.cc:477:9
    #12 0x7fa180078d32 in boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::lis
t1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator()() ../thirdparty/installed/uninstrumented/include/boo
st/bind/bind.hpp:1222:16
    #13 0x7fa180050b5b in boost::function0<void>::operator()() const ../thirdparty/installed/uninstrumented/include/
boost/function/function_template.hpp:770:14
    #14 0x7fa17d45057a in kudu::Thread::SuperviseThread(void*) ../src/kudu/util/thread.cc:657:3
    #15 0x62e0ae in __asan::AsanThread::ThreadStart(unsigned long, __sanitizer::atomic_uintptr_t*) /home/adar/Source
/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_thread.cc:259
    #16 0x7fa1816e06da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #17 0x7fa17a57388e in clone sysdeps/unix/sysv/linux/x86_64/clone.S:95
Direct leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x620470 in __interceptor_malloc /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/li
b/asan/asan_malloc_linux.cc:88
    #1 0x7fa17eba94b8 in CRYPTO_zalloc (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x1574b8)
    #2 0x7fa17eba5925  (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x153925)
    #3 0x7fa17eba60d1  (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x1540d1)
    #4 0x7fa17eb8afef in ERR_get_state (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x138fef)
    #5 0x7fa17eb8b213  (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x139213)
    #6 0x7fa17f44e28d in kudu::security::internal::ScopedCheckNoPendingSSLErrors::ScopedCheckNoPendingSSLErrors(char
 const*) ../src/kudu/security/openssl_util.h:201:5
    #7 0x7fa17f49fd51 in kudu::security::TlsSocket::Writev(iovec const*, int, long*) ../src/kudu/security/tls_socket
.cc:81:3
    #8 0x7fa18010321a in kudu::rpc::OutboundTransfer::SendBuffer(kudu::Socket&) ../src/kudu/rpc/transfer.cc:218:26
    #9 0x7fa17fff8eb6 in kudu::rpc::Connection::WriteHandler(ev::io&, int) ../src/kudu/rpc/connection.cc:791:31
    #10 0x7fa17bbaf698 in ev_invoke_pending /home/adar/Source/kudu/thirdparty/src/libev-4.20/ev.c:3155
    #11 0x7fa1800616c9 in kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) ../src/kudu/rpc/reactor.cc:176:3
    #12 0x7fa17bbb05a0 in ev_run /home/adar/Source/kudu/thirdparty/src/libev-4.20/ev.c:3555
    #13 0x7fa180061a18 in kudu::rpc::ReactorThread::RunThread() ../src/kudu/rpc/reactor.cc:477:9
    #14 0x7fa180078d32 in boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator()() ../thirdparty/installed/uninstrumented/include/boost/bind/bind.hpp:1222:16
    #15 0x7fa180050b5b in boost::function0<void>::operator()() const ../thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:770:14
    #16 0x7fa17d45057a in kudu::Thread::SuperviseThread(void*) ../src/kudu/util/thread.cc:657:3
    #17 0x62e0ae in __asan::AsanThread::ThreadStart(unsigned long, __sanitizer::atomic_uintptr_t*) /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_thread.cc:259
    #18 0x7fa1816e06da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #19 0x7fa17a57388e in clone sysdeps/unix/sysv/linux/x86_64/clone.S:95
SUMMARY: AddressSanitizer: 592 byte(s) leaked in 2 allocation(s).
../../src/kudu/tools/kudu-tool-test.cc:1540: Failure
Failed
Bad status: Runtime error: /home/adar/Source/kudu/build/asan/bin/kudu: process exited with non-zero status 1
../../src/kudu/tools/kudu-tool-test.cc:1601: Failure
Expected: RunLoadgen(1, { "--auto_database=foo", "--keep_auto_table=true" }) doesn't generate new fatal failures in the current thread.
  Actual: it does.
{noformat}

I can repro this 100% of the time on my Ubuntu 18 machine by running {{LSAN_OPTIONS=fast_unwind_on_malloc=0 bin/kudu-tool-test --gtest_filter=*LoadgenDatabaseName*}}.

It's possible this is endemic to the particular version of OpenSSL 1.1 in Ubuntu 18 (that's OpenSSL 1.1.0g).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)