You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/11/24 19:28:00 UTC
[jira] [Commented] (KUDU-2700) Memory leak in kudu-tool-test when
using OpenSSL 1.1
[ https://issues.apache.org/jira/browse/KUDU-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17238335#comment-17238335 ]
ASF subversion and git services commented on KUDU-2700:
-------------------------------------------------------
Commit b7aa01f8687d2f7f3b1478bb7824dec056ded18c in kudu's branch refs/heads/master from Grant Henke
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=b7aa01f ]
KUDU-2700: Add a LSAN suppression
OpenSSL 1.1 has a leak in libcrypto.so when running tests that use the CLI
to connect to a remote server. Previously this didn’t impact CI runs
because Jenkins and dist_test were using Ubuntu 14 and therefore
OpenSSL 1.0. Now that we are using Ubuntu 18 and OpenSSL 1.1
this leak is reproducable 100% of the time.
This patch suppresses the leak to unbreak the ASAN builds.
Change-Id: I5d1378a820997d3eabfa7e5cda53c07dd4df917d
Reviewed-on: http://gerrit.cloudera.org:8080/16781
Reviewed-by: Andrew Wong <aw...@cloudera.com>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Kudu Jenkins
> Memory leak in kudu-tool-test when using OpenSSL 1.1
> ----------------------------------------------------
>
> Key: KUDU-2700
> URL: https://issues.apache.org/jira/browse/KUDU-2700
> Project: Kudu
> Issue Type: Bug
> Components: rpc
> Affects Versions: 1.8.0
> Reporter: Adar Dembo
> Priority: Major
>
> KUDU-1889 added support for OpenSSL 1.1, but one of the outstanding issues is a memory leak reported by LSAN when running tests that use the CLI to connect to a remote server. The memory leak looks like this:
> {noformat}
> =================================================================
> ==31426==ERROR: LeakSanitizer: detected memory leaks
> Direct leak of 584 byte(s) in 1 object(s) allocated from:
> #0 0x620470 in __interceptor_malloc /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/li
> b/asan/asan_malloc_linux.cc:88
> #1 0x7fa17eba94b8 in CRYPTO_zalloc (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x1574b8)
> #2 0x7fa17eb8afdd in ERR_get_state (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x138fdd)
> #3 0x7fa17eb8b213 (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x139213)
> #4 0x7fa17f44e28d in kudu::security::internal::ScopedCheckNoPendingSSLErrors::ScopedCheckNoPendingSSLErrors(char
> const*) ../src/kudu/security/openssl_util.h:201:5
> #5 0x7fa17f49fd51 in kudu::security::TlsSocket::Writev(iovec const*, int, long*) ../src/kudu/security/tls_socket
> .cc:81:3
> #6 0x7fa18010321a in kudu::rpc::OutboundTransfer::SendBuffer(kudu::Socket&) ../src/kudu/rpc/transfer.cc:218:26
> #7 0x7fa17fff8eb6 in kudu::rpc::Connection::WriteHandler(ev::io&, int) ../src/kudu/rpc/connection.cc:791:31
> #8 0x7fa17bbaf698 in ev_invoke_pending /home/adar/Source/kudu/thirdparty/src/libev-4.20/ev.c:3155
> #9 0x7fa1800616c9 in kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) ../src/kudu/rpc/reactor.cc:176:3
> #10 0x7fa17bbb05a0 in ev_run /home/adar/Source/kudu/thirdparty/src/libev-4.20/ev.c:3555
> #11 0x7fa180061a18 in kudu::rpc::ReactorThread::RunThread() ../src/kudu/rpc/reactor.cc:477:9
> #12 0x7fa180078d32 in boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::lis
> t1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator()() ../thirdparty/installed/uninstrumented/include/boo
> st/bind/bind.hpp:1222:16
> #13 0x7fa180050b5b in boost::function0<void>::operator()() const ../thirdparty/installed/uninstrumented/include/
> boost/function/function_template.hpp:770:14
> #14 0x7fa17d45057a in kudu::Thread::SuperviseThread(void*) ../src/kudu/util/thread.cc:657:3
> #15 0x62e0ae in __asan::AsanThread::ThreadStart(unsigned long, __sanitizer::atomic_uintptr_t*) /home/adar/Source
> /kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_thread.cc:259
> #16 0x7fa1816e06da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
> #17 0x7fa17a57388e in clone sysdeps/unix/sysv/linux/x86_64/clone.S:95
> Direct leak of 8 byte(s) in 1 object(s) allocated from:
> #0 0x620470 in __interceptor_malloc /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/li
> b/asan/asan_malloc_linux.cc:88
> #1 0x7fa17eba94b8 in CRYPTO_zalloc (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x1574b8)
> #2 0x7fa17eba5925 (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x153925)
> #3 0x7fa17eba60d1 (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x1540d1)
> #4 0x7fa17eb8afef in ERR_get_state (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x138fef)
> #5 0x7fa17eb8b213 (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x139213)
> #6 0x7fa17f44e28d in kudu::security::internal::ScopedCheckNoPendingSSLErrors::ScopedCheckNoPendingSSLErrors(char
> const*) ../src/kudu/security/openssl_util.h:201:5
> #7 0x7fa17f49fd51 in kudu::security::TlsSocket::Writev(iovec const*, int, long*) ../src/kudu/security/tls_socket
> .cc:81:3
> #8 0x7fa18010321a in kudu::rpc::OutboundTransfer::SendBuffer(kudu::Socket&) ../src/kudu/rpc/transfer.cc:218:26
> #9 0x7fa17fff8eb6 in kudu::rpc::Connection::WriteHandler(ev::io&, int) ../src/kudu/rpc/connection.cc:791:31
> #10 0x7fa17bbaf698 in ev_invoke_pending /home/adar/Source/kudu/thirdparty/src/libev-4.20/ev.c:3155
> #11 0x7fa1800616c9 in kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) ../src/kudu/rpc/reactor.cc:176:3
> #12 0x7fa17bbb05a0 in ev_run /home/adar/Source/kudu/thirdparty/src/libev-4.20/ev.c:3555
> #13 0x7fa180061a18 in kudu::rpc::ReactorThread::RunThread() ../src/kudu/rpc/reactor.cc:477:9
> #14 0x7fa180078d32 in boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator()() ../thirdparty/installed/uninstrumented/include/boost/bind/bind.hpp:1222:16
> #15 0x7fa180050b5b in boost::function0<void>::operator()() const ../thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:770:14
> #16 0x7fa17d45057a in kudu::Thread::SuperviseThread(void*) ../src/kudu/util/thread.cc:657:3
> #17 0x62e0ae in __asan::AsanThread::ThreadStart(unsigned long, __sanitizer::atomic_uintptr_t*) /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_thread.cc:259
> #18 0x7fa1816e06da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
> #19 0x7fa17a57388e in clone sysdeps/unix/sysv/linux/x86_64/clone.S:95
> SUMMARY: AddressSanitizer: 592 byte(s) leaked in 2 allocation(s).
> ../../src/kudu/tools/kudu-tool-test.cc:1540: Failure
> Failed
> Bad status: Runtime error: /home/adar/Source/kudu/build/asan/bin/kudu: process exited with non-zero status 1
> ../../src/kudu/tools/kudu-tool-test.cc:1601: Failure
> Expected: RunLoadgen(1, { "--auto_database=foo", "--keep_auto_table=true" }) doesn't generate new fatal failures in the current thread.
> Actual: it does.
> {noformat}
> I can repro this 100% of the time on my Ubuntu 18 machine by running {{LSAN_OPTIONS=fast_unwind_on_malloc=0 bin/kudu-tool-test --gtest_filter=*LoadgenDatabaseName*}}.
> It's possible this is endemic to the particular version of OpenSSL 1.1 in Ubuntu 18 (that's OpenSSL 1.1.0g).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)