[jira] [Commented] (IMPALA-8924) DCHECK(!closed_) in SpillableRowBatchQueue::IsEmpty

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933711#comment-16933711 ] 

ASF subversion and git services commented on IMPALA-8924:
---------------------------------------------------------

Commit 391942d79dec18b086a85e29a75a873e74955518 in impala's branch refs/heads/master from Sahil Takiar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=391942d ]

IMPALA-8924, IMPALA-8934: Result spooling failpoint tests, fix DCHECKs

Adds several "failpoint" tests to test_result_spooling.py. These tests
use debug_actions spread throughout buffered-plan-root-sink.cc to
trigger failures while result spooling is running. The tests validate
that all queries gracefully fail and do not cause any impalad crashes.

Fixed a few bugs that came up when adding these tests, as well as the
crash reported in IMPALA-8924 (which is now covered by the failpoint
tests added in this patch).

The first bug fixed was a DCHECK in SpillableRowBatchQueue::IsEmpty()
where the method was being called after the queue had been closed. The
fix is to only call IsEmpty() if IsOpen() returns true.

The second bug was an issue in the cancellation path where
BufferedPlanRootSink::GetNext would enter an infinite loop if the query
was cancelled and then GetNext was called. The fix is to check the
cancellation state in the outer while loop.

Testing:
* Added new tests to test_result_spooling.py
* Ran core tests

Change-Id: Ib96f797bc8a5ba8baf9fb28abd1f645345bbe932
Reviewed-on: http://gerrit.cloudera.org:8080/14214
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> DCHECK(!closed_) in SpillableRowBatchQueue::IsEmpty
> ---------------------------------------------------
>
>                 Key: IMPALA-8924
>                 URL: https://issues.apache.org/jira/browse/IMPALA-8924
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>    Affects Versions: Impala 3.4.0
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>            Priority: Blocker
>
> When running exhaustive tests with result spooling enabled, there are several impalad crashes with the following stack:
> {code:java}
> #0  0x00007f5e797541f7 in raise () from /lib64/libc.so.6
> #1  0x00007f5e797558e8 in abort () from /lib64/libc.so.6
> #2  0x0000000004cc5834 in google::DumpStackTraceAndExit() ()
> #3  0x0000000004cbc28d in google::LogMessage::Fail() ()
> #4  0x0000000004cbdb32 in google::LogMessage::SendToLog() ()
> #5  0x0000000004cbbc67 in google::LogMessage::Flush() ()
> #6  0x0000000004cbf22e in google::LogMessageFatal::~LogMessageFatal() ()
> #7  0x00000000029a16cd in impala::SpillableRowBatchQueue::IsEmpty (this=0x13d504e0) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/spillable-row-batch-queue.cc:128
> #8  0x00000000025f5610 in impala::BufferedPlanRootSink::IsQueueEmpty (this=0x13943000) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/buffered-plan-root-sink.h:147
> #9  0x00000000025f4e81 in impala::BufferedPlanRootSink::GetNext (this=0x13943000, state=0x13d2a1c0, results=0x173c8520, num_results=-1, eos=0xd30cde1) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/buffered-plan-root-sink.cc:158
> #10 0x000000000294ef4d in impala::Coordinator::GetNext (this=0xe4ed180, results=0x173c8520, max_rows=-1, eos=0xd30cde1) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/coordinator.cc:683
> #11 0x0000000002251043 in impala::ClientRequestState::FetchRowsInternal (this=0xd30c800, max_rows=-1, fetched_rows=0x173c8520) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/service/client-request-state.cc:959
> #12 0x00000000022503e7 in impala::ClientRequestState::FetchRows (this=0xd30c800, max_rows=-1, fetched_rows=0x173c8520) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/service/client-request-state.cc:851
> #13 0x000000000226a36d in impala::ImpalaServer::FetchInternal (this=0x12d14800, request_state=0xd30c800, start_over=false, fetch_size=-1, query_results=0x7f5daf861138) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/service/impala-beeswax-server.cc:582
> #14 0x0000000002264970 in impala::ImpalaServer::fetch (this=0x12d14800, query_results=..., query_handle=..., start_over=false, fetch_size=-1) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/service/impala-beeswax-server.cc:188
> #15 0x00000000027caf09 in beeswax::BeeswaxServiceProcessor::process_fetch (this=0x12d6fc20, seqid=0, iprot=0x119f5780, oprot=0x119f56c0, callContext=0xdf92060) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/generated-sources/gen-cpp/BeeswaxService.cpp:3398
> #16 0x00000000027c94e6 in beeswax::BeeswaxServiceProcessor::dispatchCall (this=0x12d6fc20, iprot=0x119f5780, oprot=0x119f56c0, fname=..., seqid=0, callContext=0xdf92060) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/generated-sources/gen-cpp/BeeswaxService.cpp:3200
> #17 0x0000000002796f13 in impala::ImpalaServiceProcessor::dispatchCall (this=0x12d6fc20, iprot=0x119f5780, oprot=0x119f56c0, fname=..., seqid=0, callContext=0xdf92060) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/generated-sources/gen-cpp/ImpalaService.cpp:1824
> #18 0x0000000001b3cee4 in apache::thrift::TDispatchProcessor::process (this=0x12d6fc20, in=..., out=..., connectionContext=0xdf92060) at /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/thrift-0.9.3-p7/include/thrift/TDispatchProcessor.h:121
> #19 0x0000000001f9bf28 in apache::thrift::server::TAcceptQueueServer::Task::run (this=0xdf92000) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/rpc/TAcceptQueueServer.cpp:84
> #20 0x0000000001f9166d in impala::ThriftThread::RunRunnable (this=0x116ddfc0, runnable=..., promise=0x7f5db0862e90) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/rpc/thrift-thread.cc:74
> #21 0x0000000001f92d93 in boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long, (impala::PromiseMode)0>*>::operator() (this=0x121e7800, p=0x116ddfc0, a1=..., a2=0x7f5db0862e90) at /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/mem_fn_template.hpp:280
> #22 0x0000000001f92c29 in boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi::value<impala::Promise<unsigned long, (impala::PromiseMode)0>*> >::operator()<boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long, (impala::PromiseMode)0>*>, boost::_bi::list0> (this=0x121e7810, f=..., a=...) at /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:392
> #23 0x0000000001f92975 in boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long, (impala::PromiseMode)0>*>, boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi::value<impala::Promise<unsigned long, (impala::PromiseMode)0>*> > >::operator() (this=0x121e7800) at /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
> #24 0x0000000001f92888 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long, (impala::PromiseMode)0>*>, boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi::value<impala::Promise<unsigned long, (impala::PromiseMode)0>*> > >, void>::invoke (function_obj_ptr=...) at /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #25 0x0000000001eaf7ee in boost::function0<void>::operator() (this=0x7f5daf861ba0) at /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #26 0x0000000002401303 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) (name=..., category=..., functor=..., parent_thread_info=0x7f5db0863850, thread_started=0x7f5db0862b60) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360
> #27 0x0000000002409687 in boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) (this=0xce587c0, f=@0xce587b8: 0x2400f9c <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*)>, a=...) at /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525
> #28 0x00000000024095ab in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() (this=0xce587b8) at /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
> #29 0x000000000240956e in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run() (this=0xce58600) at /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
> #30 0x0000000003be510a in thread_proxy ()
> #31 0x00007f5e7d0bde25 in start_thread () from /lib64/libpthread.so.0
> #32 0x00007f5e7981734d in clone () from /lib64/libc.so.6 {code}
> I haven't pinned down an exact repo yet, but looking through the code it looks like this is possible during query cancellation. A query is cancelled, which eventually causes the {{SpillableRowBatchQueue}} to close as well, but a call to {{GetNext}} is still in progress, which ultimately fails because it does not check the {{closed_}} flag.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org