[jira] [Created] (IMPALA-9314) Thrift server shutdown is racey

["Sahil Takiar (Jira)" <ji...@apache.org>]

Sahil Takiar created IMPALA-9314:
------------------------------------

             Summary: Thrift server shutdown is racey
                 Key: IMPALA-9314
                 URL: https://issues.apache.org/jira/browse/IMPALA-9314
             Project: IMPALA
          Issue Type: Sub-task
          Components: Backend
            Reporter: Sahil Takiar


There are multiple issues with shutting down the {{ThriftServer}}:
 * The docs for {{ThriftServer}} say "TODO: shutdown is buggy (which only harms tests)"
 * TSAN reports a thread leak when using the {{ThriftServer}} in {{statestore-test}}
 ** Attempts to shutdown the {{ThriftServer}} in the {{Statestore}} don't seem to work properly
 * The {{TAcceptQueueServer}} actually uses a {{volatile}} {{boolean}} called {{stop_}} to coordinate the shutdown (see {{TAcceptQueueServer::stop()}}) which is not-thread safe (TSAN complains about this as well)

According to the docs this only affects test, specifically {{statestore-test}}. We should consider whether it is worth fixing the shutdown logic in the {{ThriftServer}}.
{code:java}
 WARNING: ThreadSanitizer: thread leak (pid=67752)
  Thread T39 (tid=67794, finished) created by main thread at:
    #0 pthread_create /mnt/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:889 (statestore-test+0x18903bb)
    #1 boost::thread::start_thread_noexcept() <null> (statestore-test+0x2f30b59)
    #2 boost::thread::thread<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, boost::function<void ()>, impala::ThreadDebugInfo*, impala:
:Promise<long, (impala::PromiseMode)0>*>(void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, boost::function<void ()>, impala::ThreadDebugIn
fo*, impala::Promise<long, (impala::PromiseMode)0>*) /home/systest/Impala/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:419:13 (statestore-test+0x1ebf8eb)
    #3 impala::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /home/systest/Impala/be/src/util/thread.cc:317:13 (statestore-test+0x1ebcbd5)
    #4 impala::Status impala::Thread::Create<void (impala::ThriftServer::ThriftServerEventProcessor::*)(), impala::ThriftServer::ThriftServerEventProcessor*>(std::string const&, std::string const&, void (impala::ThriftServer::ThriftServerEventProcessor::* const&)(), imp
ala::ThriftServer::ThriftServerEventProcessor* const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /home/systest/Impala/be/src/util/thread.h:81:12 (statestore-test+0x2707497)
    #5 impala::ThriftServer::ThriftServerEventProcessor::StartAndWaitForServer() /home/systest/Impala/be/src/rpc/thrift-server.cc:116:43 (statestore-test+0x27046f2)
    #6 impala::ThriftServer::Start() /home/systest/Impala/be/src/rpc/thrift-server.cc:447:60 (statestore-test+0x27063cc)
    #7 impala::Statestore::Init(int) /home/systest/Impala/be/src/statestore/statestore.cc:478:59 (statestore-test+0x1d7a431)
    #8 impala::StatestoreTest_SmokeTest_Test::TestBody() /home/systest/Impala/be/src/statestore/statestore-test.cc:55:132 (statestore-test+0x18fb381)
    #9 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (statestore-test+0x41051e2)
    #10 __libc_start_main /build/glibc-LK5gWL/glibc-2.23/csu/../csu/libc-start.c:291 (libc.so.6+0x2082f){code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)