[jira] [Commented] (IMPALA-9314) Thrift server shutdown is racey

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17065253#comment-17065253 ] 

ASF subversion and git services commented on IMPALA-9314:
---------------------------------------------------------

Commit 8b162ff480aa346d1d32830097586d681976e2d1 in impala's branch refs/heads/master from Sahil Takiar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=8b162ff ]

IMPALA-5904: (part 3) Fix more TSAN bugs

As of this commit, all backend tests are TSAN clean (or suppressions
exist for any races that are considered benign).

TSAN Fixes:
* Fixes IMPALA-9374: Possible data race in TupleDescriptor::GetLlvmStruct
    * Replaces the assignment of llvm_field_idx_ with slot_idx_, since
      they are the same; added a DCHECK to enforce this
* Data race in be/src/exprs/scalar-expr-evaluator.cc:285:26
  impala::ScalarExprEvaluator::GetValue(...)
    * This race seems specific to data-stream-test, and does not occur
      during any other tests
    * I made some changes to DataStreamTest to fix the sharing of
      TupleRowComparators between threads

Suppresions:
* Added a suppresion for IMPALA-9455: Possible data race in
  kudu::security::InitKerberosForServer
* Added suppresions for hs2-http-test since it uses the ThriftServer, which
  is known to be racey (IMPALA-9314)

Testing:
* Ran exhaustive tests
* Re-ran TSAN tests and confirmed the data races have been fixed

Change-Id: I99b7b119e256085d1ba6977e1161fc658273b242
Reviewed-on: http://gerrit.cloudera.org:8080/15363
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Tim Armstrong <ta...@cloudera.com>


> Thrift server shutdown is racey
> -------------------------------
>
>                 Key: IMPALA-9314
>                 URL: https://issues.apache.org/jira/browse/IMPALA-9314
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>            Reporter: Sahil Takiar
>            Priority: Major
>
> There are multiple issues with shutting down the {{ThriftServer}}:
>  * The docs for {{ThriftServer}} say "TODO: shutdown is buggy (which only harms tests)"
>  * TSAN reports a thread leak when using the {{ThriftServer}} in {{statestore-test}}
>  ** Attempts to shutdown the {{ThriftServer}} in the {{Statestore}} don't seem to work properly
>  * The {{TAcceptQueueServer}} actually uses a {{volatile}} {{boolean}} called {{stop_}} to coordinate the shutdown (see {{TAcceptQueueServer::stop()}}) which is not-thread safe (TSAN complains about this as well)
> According to the docs this only affects test, specifically {{statestore-test}}. We should consider whether it is worth fixing the shutdown logic in the {{ThriftServer}}.
> {code:java}
>  WARNING: ThreadSanitizer: thread leak (pid=67752)
>   Thread T39 (tid=67794, finished) created by main thread at:
>     #0 pthread_create /mnt/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:889 (statestore-test+0x18903bb)
>     #1 boost::thread::start_thread_noexcept() <null> (statestore-test+0x2f30b59)
>     #2 boost::thread::thread<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, boost::function<void ()>, impala::ThreadDebugInfo*, impala:
> :Promise<long, (impala::PromiseMode)0>*>(void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, boost::function<void ()>, impala::ThreadDebugIn
> fo*, impala::Promise<long, (impala::PromiseMode)0>*) /home/systest/Impala/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:419:13 (statestore-test+0x1ebf8eb)
>     #3 impala::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /home/systest/Impala/be/src/util/thread.cc:317:13 (statestore-test+0x1ebcbd5)
>     #4 impala::Status impala::Thread::Create<void (impala::ThriftServer::ThriftServerEventProcessor::*)(), impala::ThriftServer::ThriftServerEventProcessor*>(std::string const&, std::string const&, void (impala::ThriftServer::ThriftServerEventProcessor::* const&)(), imp
> ala::ThriftServer::ThriftServerEventProcessor* const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /home/systest/Impala/be/src/util/thread.h:81:12 (statestore-test+0x2707497)
>     #5 impala::ThriftServer::ThriftServerEventProcessor::StartAndWaitForServer() /home/systest/Impala/be/src/rpc/thrift-server.cc:116:43 (statestore-test+0x27046f2)
>     #6 impala::ThriftServer::Start() /home/systest/Impala/be/src/rpc/thrift-server.cc:447:60 (statestore-test+0x27063cc)
>     #7 impala::Statestore::Init(int) /home/systest/Impala/be/src/statestore/statestore.cc:478:59 (statestore-test+0x1d7a431)
>     #8 impala::StatestoreTest_SmokeTest_Test::TestBody() /home/systest/Impala/be/src/statestore/statestore-test.cc:55:132 (statestore-test+0x18fb381)
>     #9 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) <null> (statestore-test+0x41051e2)
>     #10 __libc_start_main /build/glibc-LK5gWL/glibc-2.23/csu/../csu/libc-start.c:291 (libc.so.6+0x2082f){code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org