You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Manikandan R (JIRA)" <ji...@apache.org> on 2019/01/03 07:38:00 UTC
[jira] [Commented] (IMPALA-7834) Impala different types of crashes
[ https://issues.apache.org/jira/browse/IMPALA-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16732727#comment-16732727 ]
Manikandan R commented on IMPALA-7834:
--------------------------------------
Had a chance to look into this?
We have seen more crashes in the past 2 weeks, mostly reoccurrences of the stack traces mentioned in JIRA description except 2 new stack traces:
15th Dec 2018:
Thread 1 (Thread 0x7f450e209700 (LWP 99538)):
#0 0x00007f4cdd56f207 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f4cdd5708f8 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00007f4cdf891185 in os::abort(bool) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#3 0x00007f4cdfa33593 in VMError::report_and_die() () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#4 0x00007f4cdf89668f in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#5 0x00007f4cdf88cbe3 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#6 <signal handler called>
No symbol table info available.
#7 0x00007f4c1ded6710 in ?? ()
No symbol table info available.
#8 0x0000000000fe74b6 in impala::PartitionedHashJoinNode::GetNext(impala::RuntimeState*, impala::RowBatch*, bool*) ()
No symbol table info available.
#9 0x0000000000fd7bb9 in impala::PartitionedAggregationNode::GetRowsStreaming(impala::RuntimeState*, impala::RowBatch*) ()
No symbol table info available.
#10 0x0000000000fd8c31 in impala::PartitionedAggregationNode::GetNext(impala::RuntimeState*, impala::RowBatch*, bool*) ()
No symbol table info available.
#11 0x0000000000b73bb2 in impala::FragmentInstanceState::ExecInternal() ()
No symbol table info available.
#12 0x0000000000b76522 in impala::FragmentInstanceState::Exec() ()
No symbol table info available.
#13 0x0000000000b65b38 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) ()
No symbol table info available.
#14 0x0000000000d16c83 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
No symbol table info available.
#15 0x0000000000d173c4 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
No symbol table info available.
#16 0x000000000128fada in thread_proxy ()
No symbol table info available.
#17 0x00007f4cdd90ddd5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#18 0x00007f4cdd637b3d in clone () from /lib64/libc.so.6
No symbol table info available.
26th Dec 2018:
Thread 1 (Thread 0x7f3063cc5700 (LWP 82059)):
#0 0x00007f394fd83207 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f394fd848f8 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00007f39520a5185 in os::abort(bool) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#3 0x00007f3952247593 in VMError::report_and_die() () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#4 0x00007f39520aa68f in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#5 0x00007f39520a0be3 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#6 <signal handler called>
No symbol table info available.
#7 0x00007f389787b1a0 in ?? ()
No symbol table info available.
#8 0x000000000102283d in impala::PhjBuilder::Send(impala::RuntimeState*, impala::RowBatch*) ()
No symbol table info available.
#9 0x000000000100dd9e in impala::Status impala::BlockingJoinNode::SendBuildInputToSink<false>(impala::RuntimeState*, impala::DataSink*) ()
No symbol table info available.
#10 0x000000000100c509 in impala::BlockingJoinNode::ProcessBuildInputAndOpenProbe(impala::RuntimeState*, impala::DataSink*) ()
No symbol table info available.
#11 0x0000000000fe83cf in impala::PartitionedHashJoinNode::Open(impala::RuntimeState*) ()
No symbol table info available.
#12 0x0000000000b74d6d in impala::FragmentInstanceState::Open() ()
No symbol table info available.
#13 0x0000000000b763ab in impala::FragmentInstanceState::Exec() ()
No symbol table info available.
#14 0x0000000000b65b38 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) ()
No symbol table info available.
#15 0x0000000000d16c83 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
No symbol table info available.
#16 0x0000000000d173c4 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
No symbol table info available.
#17 0x000000000128fada in thread_proxy ()
No symbol table info available.
#18 0x00007f3950121dd5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#19 0x00007f394fe4bb3d in clone () from /lib64/libc.so.6
No symbol table info available.
> Impala different types of crashes
> ---------------------------------
>
> Key: IMPALA-7834
> URL: https://issues.apache.org/jira/browse/IMPALA-7834
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.11.0
> Reporter: Manikandan R
> Priority: Critical
> Labels: CDH5, crash
> Attachments: stacktrace_106154_25Nov2018.txt, stacktrace_10917_19Nov2018.txt, stacktrace_117834_06Nov2018.txt, stacktrace_117834_06Nov2018.txt, stacktrace_119889_18Nov2018.txt, stacktrace_12121_22Nov2018.txt, stacktrace_122223_14Nov2018.txt, stacktrace_125057_05Nov.txt, stacktrace_14175_23Nov2018.txt, stacktrace_15618_20Nov2018.txt, stacktrace_17249_20Nov2018.txt, stacktrace_24446_27Nov2018.txt, stacktrace_28839_23Nov2018.txt, stacktrace_29716_21Nov2018.txt, stacktrace_59011_19Nov2018.txt, stacktrace_65470_23Nov2018.txt, stacktrace_72492_01Nov2018.txt, stacktrace_72492_01Nov2018.txt, stacktrace_74831_28Oct.txt, stacktrace_8486_14Nov2018.txt, stacktrace_84892_26Nov2018.txt
>
>
> Off late, We had witnessed different types of crashes in cluster. I don't see any similarities among crashes stack traces and also not able to reproduce. Below are the stack traces occurred in different daemons at different timings. I do have complete stack traces and let me know if it helps for further debugging.
> 1)
> 10-1-33-172
> Nov 6 18:04
> Thread 1 (Thread 0x7f018b423700 (LWP 96325)):
> #0 0x00007f0aaaf5e207 in raise () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f0aaaf5fa38 in abort () from /lib64/libc.so.6
> No symbol table info available.
> #2 0x00007f0aad280185 in os::abort(bool) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #3 0x00007f0aad422593 in VMError::report_and_die() () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #4 0x00007f0aad28568f in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #5 0x00007f0aad27bbe3 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #6 <signal handler called>
> No symbol table info available.
> #7 0x00007f0a44ca3000 in ?? ()
> No symbol table info available.
> #8 0x0000000000dba5c1 in impala::HdfsParquetScanner::TransferScratchTuples(impala::RowBatch*) ()
> No symbol table info available.
> #9 0x0000000000dba924 in impala::HdfsParquetScanner::AssembleRows(std::vector<impala::ParquetColumnReader*, std::allocator<impala::ParquetColumnReader*> > const&, impala::RowBatch*, bool*) ()
> No symbol table info available.
> #10 0x0000000000dbf5f6 in impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) ()
> No symbol table info available.
> #11 0x0000000000db9ba7 in impala::HdfsParquetScanner::ProcessSplit() ()
> No symbol table info available.
> #12 0x0000000000d835e6 in impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::MemPool*, impala::io::ScanRange*) ()
> No symbol table info available.
> #13 0x0000000000d85115 in impala::HdfsScanNode::ScannerThread() ()
> No symbol table info available.
> #14 0x0000000000d16c83 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
> No symbol table info available.
> #15 0x0000000000d173c4 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
> No symbol table info available.
> #16 0x000000000128fada in thread_proxy ()
> No symbol table info available.
> #17 0x00007f0aab2fcdd5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #18 0x00007f0aab026b3d in clone () from /lib64/libc.so.6
> No symbol table info available.
> Nov 1 11:21
> Thread 1 (Thread 0x7fb3bffe9700 (LWP 50334)):
> #0 0x00007fc2959f0207 in raise () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007fc2959f18f8 in abort () from /lib64/libc.so.6
> No symbol table info available.
> #2 0x00007fc297d12185 in os::abort(bool) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #3 0x00007fc297eb4593 in VMError::report_and_die() () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #4 0x00007fc297d1768f in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #5 0x00007fc297d0dbe3 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #6 <signal handler called>
> No symbol table info available.
> #7 0x00007fc1dd9f0000 in ?? ()
> No symbol table info available.
> #8 0x0000000000fdd9f9 in impala::PartitionedAggregationNode::Open(impala::RuntimeState*) ()
> No symbol table info available.
> #9 0x0000000000b74d6d in impala::FragmentInstanceState::Open() ()
> No symbol table info available.
> #10 0x0000000000b763ab in impala::FragmentInstanceState::Exec() ()
> No symbol table info available.
> #11 0x0000000000b65b38 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) ()
> No symbol table info available.
> #12 0x0000000000d16c83 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
> No symbol table info available.
> #13 0x0000000000d173c4 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
> No symbol table info available.
> #14 0x000000000128fada in thread_proxy ()
> No symbol table info available.
> #15 0x00007fc295d8edd5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #16 0x00007fc295ab8b3d in clone () from /lib64/libc.so.6
> No symbol table info available.
> 2)
> 10-1-42-100
> Oct 28 07:15
> (I am seeing this particular issue being discussed in https://issues.apache.org/jira/browse/IMPALA-7194)
> Thread 1 (Thread 0x7f6ef14f6700 (LWP 15999)):
> #0 0x00007f7c95316207 in raise () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f7c953178f8 in abort () from /lib64/libc.so.6
> No symbol table info available.
> #2 0x00007f7c97638185 in os::abort(bool) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #3 0x00007f7c977da593 in VMError::report_and_die() () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #4 0x00007f7c9763d68f in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #5 0x00007f7c97633be3 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #6 <signal handler called>
> No symbol table info available.
> #7 0x00007f7bdcdaa830 in ?? ()
> No symbol table info available.
> #8 0x0000000000fddd0f in impala::PartitionedAggregationNode::Open(impala::RuntimeState*) ()
> No symbol table info available.
> #9 0x0000000000b74d6d in impala::FragmentInstanceState::Open() ()
> No symbol table info available.
> #10 0x0000000000b763ab in impala::FragmentInstanceState::Exec() ()
> No symbol table info available.
> #11 0x0000000000b65b38 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) ()
> No symbol table info available.
> #12 0x0000000000d16c83 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
> No symbol table info available.
> #13 0x0000000000d173c4 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
> No symbol table info available.
> #14 0x000000000128fada in thread_proxy ()
> No symbol table info available.
> #15 0x00007f7c956b4dd5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #16 0x00007f7c953deb3d in clone () from /lib64/libc.so.6
> No symbol table info available.
> 3)
> 10-1-43-65
> Nov 5 19:25
> Thread 1 (Thread 0x7f484dfc3700 (LWP 5561)):
> #0 0x00007f4c61734207 in raise () from /lib64/libc.so.6
> No symbol table info available.
> #1 0x00007f4c617358f8 in abort () from /lib64/libc.so.6
> No symbol table info available.
> #2 0x00007f4c63a56185 in os::abort(bool) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #3 0x00007f4c63bf8593 in VMError::report_and_die() () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #4 0x00007f4c63a5b68f in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #5 0x00007f4c63a51be3 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> No symbol table info available.
> #6 <signal handler called>
> No symbol table info available.
> #7 0x00007f4ba9146250 in ?? ()
> No symbol table info available.
> #8 0x0000000001023b52 in impala::PhjBuilder::Partition::BuildHashTable(bool*) ()
> No symbol table info available.
> #9 0x0000000001023dde in impala::PhjBuilder::BuildHashTablesAndPrepareProbeStreams() ()
> No symbol table info available.
> #10 0x0000000001024340 in impala::PhjBuilder::FlushFinal(impala::RuntimeState*) ()
> No symbol table info available.
> #11 0x000000000100da6d in impala::Status impala::BlockingJoinNode::SendBuildInputToSink<true>(impala::RuntimeState*, impala::DataSink*) ()
> No symbol table info available.
> #12 0x000000000100c2a0 in impala::BlockingJoinNode::ProcessBuildInputAsync(impala::RuntimeState*, impala::DataSink*, impala::Status*) ()
> No symbol table info available.
> #13 0x0000000000d16c83 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
> No symbol table info available.
> #14 0x0000000000d173c4 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
> No symbol table info available.
> #15 0x000000000128fada in thread_proxy ()
> No symbol table info available.
> #16 0x00007f4c61ad2dd5 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #17 0x00007f4c617fcb3d in clone () from /lib64/libc.so.6
> No symbol table info available
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org