You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Adar Dembo (JIRA)" <ji...@apache.org> on 2018/05/15 22:07:00 UTC
[jira] [Commented] (KUDU-2439) There's no way to safely clean up a
KuduClient or Messenger
[ https://issues.apache.org/jira/browse/KUDU-2439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476536#comment-16476536 ]
Adar Dembo commented on KUDU-2439:
----------------------------------
Note that with the suppression removed, the data race can actually result in a crash:
{noformat}
[ OK ] MultiDirClusterITest.TestBasicMultiDirCluster (4882 ms)
[----------] 1 test from MultiDirClusterITest (4882 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (4883 ms total)
[ PASSED ] 1 test.
W0515 15:01:01.031991 7264 connection.cc:512] client connection to 127.7.11.254:35729 recv error: Network error: failed to read from TLS socket (remote: 127.7.11.254:35729): Connection reset by peer (error 104)
W0515 15:01:01.031992 8381 connection.cc:512] client connection to 127.7.11.254:35729 recv error: Network error: failed to read from TLS socket (remote: 127.7.11.254:35729): Connection reset by peer (error 104)
*** Aborted at 1526421661 (unix time) try "date -d @1526421661" if you are using GNU date ***
PC: @ 0x7f74d0e1c18a __GI___pthread_rwlock_unlock
*** SIGSEGV (@0x18) received by PID 7215 (TID 0x7f74c63ff700) from PID 24; stack trace: ***
@ 0x456280 __tsan::CallUserSignalHandler() at /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1908
@ 0x45801e rtl_sigaction() at /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1997
@ 0x7f74d0e21890 (unknown) at ??:0
@ 0x7f74d0e1c18a __GI___pthread_rwlock_unlock at ??:0
@ 0x47a5c2 pthread_rwlock_unlock at /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1385
@ 0x7f74ce6040b5 CRYPTO_THREAD_unlock at ??:0
@ 0x7f74ce59446b CRYPTO_free_ex_data at ??:0
@ 0x7f74ce1aa241 SSL_free at ??:0
@ 0x7f74d045b89e _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRPFvP6ssl_stES4_EEEvDpOT_ at ??:0
@ 0x7f74d045b7e5 std::__1::__function::__func<>::operator()() at ??:0
@ 0x7f74d428274e std::__1::function<>::operator()() at ??:0
@ 0x7f74d045deca kudu::security::TlsSocket::Close() at ??:0
@ 0x7f74d42887df kudu::rpc::Connection::Shutdown() at ??:0
@ 0x7f74d42b2643 kudu::rpc::ReactorThread::DestroyConnection() at ??:0
@ 0x7f74d428a385 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7f74d42914bb ev::base<>::method_thunk<>() at ??:0
@ 0x7f74cf68b741 ev_invoke_pending at ??:0
@ 0x7f74d42b0d66 kudu::rpc::ReactorThread::InvokePendingCb() at ??:0
@ 0x7f74cf68c84e ev_run at ??:0
@ 0x7f74d42b69c9 ev::loop_ref::run() at ??:0
@ 0x7f74d42b0eab kudu::rpc::ReactorThread::RunThread() at ??:0
@ 0x7f74d42bac6a boost::_mfi::mf0<>::operator()() at ??:0
@ 0x7f74d42babbb boost::_bi::list1<>::operator()<>() at ??:0
@ 0x7f74d42bab44 boost::_bi::bind_t<>::operator()() at ??:0
@ 0x7f74d42ba93a boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0
@ 0x7f74d42aaa22 boost::function0<>::operator()() at ??:0
@ 0x7f74d0320445 kudu::Thread::SuperviseThread() at ??:0
@ 0x45615e __tsan_thread_start_func at /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:985
@ 0x7f74d0e166db start_thread at ??:0
@ 0x7f74cd73688f clone at ??:0
{noformat}
> There's no way to safely clean up a KuduClient or Messenger
> -----------------------------------------------------------
>
> Key: KUDU-2439
> URL: https://issues.apache.org/jira/browse/KUDU-2439
> Project: Kudu
> Issue Type: Bug
> Components: client, rpc
> Affects Versions: 1.8.0
> Reporter: Adar Dembo
> Priority: Major
>
> KuduClient has shared ownership, and its only "shutdown" knob is to drop its last ref. This drops the last ref on the underlying Messenger object, but Messenger itself has a funky "internal" vs. "external" ref system, and destroying the KuduClient only drops the last external ref. The Messenger is only destroyed when the last internal ref is dropped, and that only happens when outstanding reactor threads finish whatever processing they were busy doing. So, there's no way for a user to "destroy this KuduClient and wait for all outstanding resources to be cleaned up".
> Why is this important? For one, there's a known data race with outstanding work done by the KuduClient's DnsResolver. For two, there's a similar issue with OpenSSL. OpenSSL 1.1 registers an atexit() handler that tears down its global library state. For this to be safe, all allocated OpenSSL resources must have been released by the time atexit() handlers run (which is to say, all OpenSSL resources must be released by the time main() returns or exit() is called). Because we can't wait on the KuduClient to destroy itself and its OpenSSL resources, applications using the KuduClient may run the atexit() handler at an unsafe time.
> Here's the TSAN output for the OpenSSL data race:. It's trivial to reproduce via reactor-test once the appropriate suppression is removed from tsan-suppressions.txt:
> {noformat}
> WARNING: ThreadSanitizer: data race (pid=7914)
> Write of size 1 at 0x7b1000004340 by main thread:
> #0 pthread_rwlock_destroy /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1313 (reactor-test+0x48205e)
> #1 CRYPTO_THREAD_lock_free /home/adar/openssl/openssl-1.1.0g/build_shared/../crypto/threads_pthread.c:95 (libcrypto.so.1.1+0x20d0e5)
> Previous atomic read of size 1 at 0x7b1000004340 by thread T16:
> #0 pthread_rwlock_wrlock /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1352 (reactor-test+0x45ffe3)
> #1 CRYPTO_THREAD_write_lock /home/adar/openssl/openssl-1.1.0g/build_shared/../crypto/threads_pthread.c:66 (libcrypto.so.1.1+0x20d08a)
> #2 std::__1::__function::__func<void (*)(ssl_ctx_st*), std::__1::allocator<void (*)(ssl_ctx_st*)>, void (ssl_ctx_st*)>::operator()(ssl_ctx_st*&&) /home/adar/Source/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1562:12 (libsecurity.so+0x56b14)
> #3 std::__1::function<void (ssl_ctx_st*)>::operator()(ssl_ctx_st*) const /home/adar/Source/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1916:12 (libkrpc.so+0xb139d)
> #4 std::__1::unique_ptr<ssl_ctx_st, std::__1::function<void (ssl_ctx_st*)> >::reset(ssl_ctx_st*) /home/adar/Source/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2598:7 (libkrpc.so+0xb0f9e)
> #5 std::__1::unique_ptr<ssl_ctx_st, std::__1::function<void (ssl_ctx_st*)> >::~unique_ptr() /home/adar/Source/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2552 (libkrpc.so+0xb0f9e)
> #6 kudu::security::TlsContext::~TlsContext() /home/adar/Source/kudu/build/tsan/../../src/kudu/security/tls_context.h:77 (libkrpc.so+0xb0f9e)
> #7 std::__1::default_delete<kudu::security::TlsContext>::operator()(kudu::security::TlsContext*) const /home/adar/Source/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2285:5 (libkrpc.so+0xab32c)
> #8 std::__1::unique_ptr<kudu::security::TlsContext, std::__1::default_delete<kudu::security::TlsContext> >::reset(kudu::security::TlsContext*) /home/adar/Source/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2598 (libkrpc.so+0xab32c)
> #9 std::__1::unique_ptr<kudu::security::TlsContext, std::__1::default_delete<kudu::security::TlsContext> >::~unique_ptr() /home/adar/Source/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2552 (libkrpc.so+0xab32c)
> #10 kudu::rpc::Messenger::~Messenger() /home/adar/Source/kudu/build/tsan/../../src/kudu/rpc/messenger.cc:433 (libkrpc.so+0xab32c)
> #11 std::__1::default_delete<kudu::rpc::Messenger>::operator()(kudu::rpc::Messenger*) const /home/adar/Source/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2285:5 (libkrpc.so+0xb0ddb)
> #12 std::__1::__shared_ptr_pointer<kudu::rpc::Messenger*, std::__1::default_delete<kudu::rpc::Messenger>, std::__1::allocator<kudu::rpc::Messenger> >::__on_zero_shared() /home/adar/Source/kudu/thirdparty/installed/tsan/include/c++/v1/memory:3586 (libkrpc.so+0xb0ddb)
> #13 std::__1::__shared_count::__release_shared() /home/adar/Source/kudu/thirdparty/installed/tsan/include/c++/v1/memory:3490:9 (reactor-test+0x4d0a2e)
> #14 std::__1::__shared_weak_count::__release_shared() /home/adar/Source/kudu/thirdparty/installed/tsan/include/c++/v1/memory:3532 (reactor-test+0x4d0a2e)
> #15 std::__1::shared_ptr<kudu::rpc::Messenger>::~shared_ptr() /home/adar/Source/kudu/thirdparty/installed/tsan/include/c++/v1/memory:4468 (reactor-test+0x4d0a2e)
> #16 std::__1::shared_ptr<kudu::rpc::Messenger>::reset() /home/adar/Source/kudu/thirdparty/installed/tsan/include/c++/v1/memory:4603:5 (libkrpc.so+0xbefd1)
> #17 kudu::rpc::ReactorThread::RunThread() /home/adar/Source/kudu/build/tsan/../../src/kudu/rpc/reactor.cc:482 (libkrpc.so+0xbefd1)
> #18 boost::_mfi::mf0<void, kudu::rpc::ReactorThread>::operator()(kudu::rpc::ReactorThread*) const /home/adar/Source/kudu/build/tsan/../../thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc8c69)
> #19 void boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> >::operator()<boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>&, boost::_bi::list0&, int) /home/adar/Source/kudu/build/tsan/../../thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc8bba)
> #20 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator()() /home/adar/Source/kudu/build/tsan/../../thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc8b43)
> #21 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >, void>::invoke(boost::detail::function::function_buffer&) /home/adar/Source/kudu/build/tsan/../../thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc8939)
> #22 boost::function0<void>::operator()() const /home/adar/Source/kudu/build/tsan/../../thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb8a21)
> #23 kudu::Thread::SuperviseThread(void*) /home/adar/Source/kudu/build/tsan/../../src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1c1264)
> Location is heap block of size 56 at 0x7b1000004340 allocated by main thread:
> #0 malloc /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:666 (reactor-test+0x47cc9c)
> #1 CRYPTO_malloc /home/adar/openssl/openssl-1.1.0g/build_shared/../crypto/mem.c:92 (libcrypto.so.1.1+0x1a6327)
> #2 CRYPTO_THREAD_run_once /home/adar/openssl/openssl-1.1.0g/build_shared/../crypto/threads_pthread.c:106 (libcrypto.so.1.1+0x20d126)
> #3 CRYPTO_THREAD_run_once /home/adar/openssl/openssl-1.1.0g/build_shared/../crypto/threads_pthread.c:106 (libcrypto.so.1.1+0x20d126)
> #4 kudu::rpc::Messenger::Init() /home/adar/Source/kudu/build/tsan/../../src/kudu/rpc/messenger.cc:444:3 (libkrpc.so+0xa9589)
> #5 kudu::rpc::MessengerBuilder::Build(std::__1::shared_ptr<kudu::rpc::Messenger>*) /home/adar/Source/kudu/build/tsan/../../src/kudu/rpc/messenger.cc:205:3 (libkrpc.so+0xa903d)
> #6 kudu::rpc::RpcTestBase::CreateMessenger(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<kudu::rpc::Messenger>*, int, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /home/adar/Source/kudu/build/tsan/../../src/kudu/rpc/rpc-test-base.h:462:16 (reactor-test+0x4d3412)
> #7 kudu::rpc::ReactorTest::SetUp() /home/adar/Source/kudu/build/tsan/../../src/kudu/rpc/reactor-test.cc:46:5 (reactor-test+0x4d02ae)
> #8 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x551df)
> #9 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x551df)
> #10 testing::Test::Run() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2470:3 (libgmock.so+0x342b1)
> #11 testing::TestInfo::Run() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3563c)
> #12 testing::TestCase::Run() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36116)
> #13 testing::internal::UnitTestImpl::RunAllTests() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x424ea)
> #14 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5614f)
> #15 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5614f)
> #16 testing::UnitTest::Run() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41dd2)
> #17 RUN_ALL_TESTS() /home/adar/Source/kudu/build/tsan/../../thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33fb)
> #18 main /home/adar/Source/kudu/build/tsan/../../src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2bb6)
> Thread T16 'rpc reactor-793' (tid=7934, finished) created by main thread at:
> #0 pthread_create /home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (reactor-test+0x45f7d6)
> #1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::function<void ()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /home/adar/Source/kudu/build/tsan/../../src/kudu/util/thread.cc:556:15 (libkudu_util.so+0x1c0c8f)
> #2 kudu::Status kudu::Thread::Create<void (kudu::rpc::ReactorThread::*)(), kudu::rpc::ReactorThread*>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, void (kudu::rpc::ReactorThread::* const&)(), kudu::rpc::ReactorThread* const&, scoped_refptr<kudu::Thread>*) /home/adar/Source/kudu/build/tsan/../../src/kudu/util/thread.h:164:12 (libkrpc.so+0xc4095)
> #3 kudu::rpc::ReactorThread::Init() /home/adar/Source/kudu/build/tsan/../../src/kudu/rpc/reactor.cc:168:10 (libkrpc.so+0xbeace)
> #4 kudu::rpc::Reactor::Init() /home/adar/Source/kudu/build/tsan/../../src/kudu/rpc/reactor.cc:728:18 (libkrpc.so+0xc2f81)
> #5 kudu::rpc::Messenger::Init() /home/adar/Source/kudu/build/tsan/../../src/kudu/rpc/messenger.cc:446:5 (libkrpc.so+0xa95e2)
> #6 kudu::rpc::MessengerBuilder::Build(std::__1::shared_ptr<kudu::rpc::Messenger>*) /home/adar/Source/kudu/build/tsan/../../src/kudu/rpc/messenger.cc:205:3 (libkrpc.so+0xa903d)
> #7 kudu::rpc::RpcTestBase::CreateMessenger(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<kudu::rpc::Messenger>*, int, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /home/adar/Source/kudu/build/tsan/../../src/kudu/rpc/rpc-test-base.h:462:16 (reactor-test+0x4d3412)
> #8 kudu::rpc::ReactorTest::SetUp() /home/adar/Source/kudu/build/tsan/../../src/kudu/rpc/reactor-test.cc:46:5 (reactor-test+0x4d02ae)
> #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x551df)
> #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x551df)
> #11 testing::Test::Run() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2470:3 (libgmock.so+0x342b1)
> #12 testing::TestInfo::Run() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3563c)
> #13 testing::TestCase::Run() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36116)
> #14 testing::internal::UnitTestImpl::RunAllTests() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x424ea)
> #15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5614f)
> #16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5614f)
> #17 testing::UnitTest::Run() /home/adar/Source/kudu/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41dd2)
> #18 RUN_ALL_TESTS() /home/adar/Source/kudu/build/tsan/../../thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33fb)
> #19 main /home/adar/Source/kudu/build/tsan/../../src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2bb6)
> SUMMARY: ThreadSanitizer: data race /home/adar/openssl/openssl-1.1.0g/build_shared/../crypto/threads_pthread.c:95 in CRYPTO_THREAD_lock_free
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)