You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tim Armstrong (Jira)" <ji...@apache.org> on 2020/02/21 00:50:00 UTC
[jira] [Resolved] (IMPALA-9407) Impalad crash in
RuntimeProfile::EventSequence
[ https://issues.apache.org/jira/browse/IMPALA-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-9407.
-----------------------------------
Resolution: Duplicate
This looks like our old friend IMPALA-4631. This happened on a CentOS 6.4 vm on a m2-4xlarge ec2 instance.
{noformat}
F0217 02:10:47.921636 14885 runtime-profile-counters.h:572] 9945dfcfab2eb54c:1609788600000001] Check failed: offset_ >= -3 (-15 vs. -3)
{noformat}
> Impalad crash in RuntimeProfile::EventSequence
> ----------------------------------------------
>
> Key: IMPALA-9407
> URL: https://issues.apache.org/jira/browse/IMPALA-9407
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 3.4.0
> Reporter: Attila Jeges
> Assignee: Tim Armstrong
> Priority: Blocker
> Labels: broken-build, flaky
>
> impalad instance crashed during exhaustive tests, probably while running TestExprs.test_exprs:
> {code:java}
> 18:00:32 TestExprs.test_exprs[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: orc/def/block | enable_expr_rewrites: 0]
> 18:00:32 [gw1] linux2 -- Python 2.6.6 /data/jenkins/workspace/impala-asf-master-exhaustive-centos6-shard2/repos/Impala/bin/../infra/python/env/bin/python
> 18:00:32 query_test/test_exprs.py:60: in test_exprs
> 18:00:32 self.run_test_case('QueryTest/exprs', vector)
> 18:00:32 common/impala_test_suite.py:659: in run_test_case
> 18:00:32 result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> 18:00:32 common/impala_test_suite.py:594: in __exec_in_impala
> 18:00:32 result = self.__execute_query(target_impalad_client, query, user=user)
> 18:00:32 common/impala_test_suite.py:933: in __execute_query
> 18:00:32 return impalad_client.execute(query, user=user)
> 18:00:32 common/impala_connection.py:205: in execute
> 18:00:32 return self.__beeswax_client.execute(sql_stmt, user=user)
> 18:00:32 beeswax/impala_beeswax.py:187: in execute
> 18:00:32 handle = self.__execute_query(query_string.strip(), user=user)
> 18:00:32 beeswax/impala_beeswax.py:365: in __execute_query
> 18:00:32 self.wait_for_finished(handle)
> 18:00:32 beeswax/impala_beeswax.py:386: in wait_for_finished
> 18:00:32 raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> 18:00:32 E ImpalaBeeswaxException: ImpalaBeeswaxException:
> 18:00:32 E Query aborted:Failed due to unreachable impalad(s): shared-centos64-ec2-m2-4xlarge-ondemand-13c1.vpc.cloudera.com:22002
> {code}
> impalad minidump:
> {code:java}
> Operating system: Linux
> 0.0.0 Linux 2.6.32-358.14.1.el6.centos.plus.x86_64 #1 SMP Tue Jul 16 21:33:24 UTC 2013 x86_64
> CPU: amd64
> family 6 model 45 stepping 7
> 8 CPUs
> GPU: UNKNOWN
> Crash reason: SIGABRT
> Crash address: 0x45200004ed0
> Process uptime: not available
> Thread 373 (crashed)
> 0 libc-2.12.so + 0x328e5
> rax = 0x0000000000000000 rdx = 0x0000000000000006
> rcx = 0xffffffffffffffff rbx = 0x00000000072153c0
> rsi = 0x0000000000003a25 rdi = 0x0000000000004ed0
> rbp = 0x00007f2b804a9de0 rsp = 0x00007f2b804a9a78
> r8 = 0x00007f2b804a9b00 r9 = 0x0000000000003a25
> r10 = 0x0000000000000008 r11 = 0x0000000000000202
> r12 = 0x0000000007215440 r13 = 0x000000000000004c
> r14 = 0x000000000721cd84 r15 = 0x00000000072153c0
> rip = 0x00000034f4c328e5
> Found by: given as instruction pointer in context
> 1 libc-2.12.so + 0x340c5
> rbp = 0x00007f2b804a9de0 rsp = 0x00007f2b804a9a80
> rip = 0x00000034f4c340c5
> Found by: stack scanning
> 2 impalad!boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, void>::invoke [function_template.hpp : 153 + 0xb]
> rbp = 0x00007f2b804a9de0 rsp = 0x00007f2b804a9a88
> rip = 0x00000000021b94da
> Found by: stack scanning
> 3 impalad!google::LogMessage::Flush() + 0x157
> rbp = 0x00007f2c83fb27a0 rsp = 0x00007f2b804a9df0
> rip = 0x0000000004e4e3b7
> Found by: call frame info
> 4 impalad!google::LogMessageFatal::~LogMessageFatal() + 0xe
> rbx = 0x00007f2b804a9eb0 rbp = 0x00007f2b804a9ed0
> rsp = 0x00007f2b804a9e70 r12 = 0x0000000000000001
> r13 = 0x0000000000000001 r14 = 0x0000000000000001
> r15 = 0x0000000000000001 rip = 0x0000000004e5197e
> Found by: call frame info
> 5 impalad!impala::RuntimeProfile::EventSequence::Start(long) [runtime-profile-counters.h : 572 + 0xc]
> rbx = 0x000000000caaee00 rbp = 0x00007f2b804a9ed0
> rsp = 0x00007f2b804a9e90 r12 = 0x0000000000000001
> r13 = 0x0000000000000001 r14 = 0x0000000000000001
> r15 = 0x0000000000000001 rip = 0x00000000021aa49c
> Found by: call frame info
> 6 impalad!impala::FragmentInstanceState::Prepare() [fragment-instance-state.cc : 153 + 0x2e]
> rbx = 0x000000000caaee00 rbp = 0x00007f2b804aa4f0
> rsp = 0x00007f2b804a9ee0 r12 = 0x0000000000000001
> r13 = 0x0000000000000001 r14 = 0x0000000000000001
> r15 = 0x0000000000000001 rip = 0x00000000021a42f8
> Found by: call frame info
> 7 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc : 75 + 0x19]
> rbx = 0x0000000000000001 rbp = 0x00007f2b804aa650
> rsp = 0x00007f2b804aa500 r12 = 0x0000000000000001
> r13 = 0x0000000000000001 r14 = 0x0000000000000001
> r15 = 0x0000000000000001 rip = 0x00000000021a366e
> Found by: call frame info
> 8 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) [query-state.cc : 694 + 0x19]
> rbx = 0x0000000000000001 rbp = 0x00007f2b804aa760
> rsp = 0x00007f2b804aa660 r12 = 0x0000000000000001
> r13 = 0x0000000000000001 r14 = 0x0000000000000001
> r15 = 0x0000000000000001 rip = 0x00000000021b78f3
> Found by: call frame info
> 9 impalad!impala::QueryState::StartFInstances()::{lambda()#1}::operator()() const + 0x26
> rbx = 0x0000000000003a25 rbp = 0x00007f2b804aa780
> rsp = 0x00007f2b804aa770 r12 = 0x0000000000000000
> r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8
> r15 = 0x0000000000000007 rip = 0x00000000021b5c12
> Found by: call frame info
> 10 impalad!boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, void>::invoke [function_template.hpp : 153 + 0xc]
> rbx = 0x0000000000003a25 rbp = 0x00007f2b804aa7b0
> rsp = 0x00007f2b804aa790 r12 = 0x0000000000000000
> r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8
> r15 = 0x0000000000000007 rip = 0x00000000021b94db
> Found by: call frame info
> 11 impalad!boost::function0<void>::operator()() const [function_template.hpp : 767 + 0x11]
> rbx = 0x0000000000003a25 rbp = 0x00007f2b804aa7f0
> rsp = 0x00007f2b804aa7c0 r12 = 0x0000000000000000
> r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8
> r15 = 0x0000000000000007 rip = 0x0000000001f9c39e
> Found by: call frame info
> 12 impalad!impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) [thread.cc : 360 + 0xf]
> rbx = 0x0000000000003a25 rbp = 0x00007f2b804aab50
> rsp = 0x00007f2b804aa800 r12 = 0x0000000000000000
> r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8
> r15 = 0x0000000000000007 rip = 0x000000000255ac29
> Found by: call frame info
> 13 impalad!void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) [bind.hpp : 525 + 0x15]
> rbx = 0x000000000255a8c2 rbp = 0x00007f2b804aabc0
> rsp = 0x00007f2b804aab60 r12 = 0x00007f2b80eab830
> r13 = 0x00007f2b80eaa7d0 r14 = 0x00000000177065c8
> r15 = 0x0000000000000007 rip = 0x0000000002562fad
> Found by: call frame info
> 14 impalad!boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() [bind_template.hpp : 20 + 0x22]
> rbx = 0x0000000000000000 rbp = 0x00007f2b804aac10
> rsp = 0x00007f2b804aabd0 r12 = 0x00000034f5011200
> r13 = 0x00007f2b804ab9c0 r14 = 0x0000000000000004
> r15 = 0x0000000000000007 rip = 0x0000000002562ed1
> Found by: call frame info
> 15 impalad!boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run() [thread.hpp : 116 + 0x12]
> rbx = 0x0000000000000000 rbp = 0x00007f2b804aac30
> rsp = 0x00007f2b804aac20 r12 = 0x00000034f5011200
> r13 = 0x00007f2b804ab9c0 r14 = 0x0000000000000004
> r15 = 0x0000000000000007 rip = 0x0000000002562e94
> Found by: call frame info
> 16 impalad!thread_proxy + 0xda
> rbx = 0x0000000000000000 rbp = 0x0000000000000000
> rsp = 0x00007f2b804aac40 r12 = 0x00000034f5011200
> r13 = 0x00007f2b804ab9c0 r14 = 0x0000000000000004
> r15 = 0x0000000000000007 rip = 0x0000000003d77d5a
> Found by: call frame info
> 17 libpthread-2.12.so + 0x7851
> rbx = 0x0000000000000000 rbp = 0x0000000000000000
> rsp = 0x00007f2b804aac80 r12 = 0x00000034f5011200
> r13 = 0x00007f2b804ab9c0 r14 = 0x0000000000000004
> r15 = 0x0000000000000007 rip = 0x00000034f5007851
> Found by: call frame info
> 18 libpthread-2.12.so + 0x11200
> rsp = 0x00007f2b804aad28 rip = 0x00000034f5011200
> Found by: stack scanning
> 19 libpthread-2.12.so + 0x11200
> rsp = 0x00007f2b804aada0 rip = 0x00000034f5011200
> Found by: stack scanning
> 20 libc-2.12.so + 0xe894d
> rsp = 0x00007f2b804aadc0 rip = 0x00000034f4ce894d
> Found by: stack scanning
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)