Re: Impalad crash

[Casey Ching <ca...@cloudera.com>]

Hi Nick,

We seem to have found the problem. I filed https://issues.cloudera.org/browse/IMPALA-3105 about it. The query leads to memory corruption so it’s best to avoid it. I couldn’t reproduce the problem on Impala’s trunk. I expect this will be fixed in the 0.8 Kudu/Impala release. Let me know if you have questions and thanks again for helping with this.

Casey

On February 26, 2016 at 3:36:18 PM, Casey Ching (casey@cloudera.com) wrote:

Thanks for sending this Nick. Unfortunately it looks like a different issue than the previous one — impala::RawValue::Write(void const*, void*, impala::ColumnType const&, impala::MemPool*)

I looked at ExprContext::GetValue and didn’t see anything obviously wrong. Would you be wiling to upload the core dump? They typically compress well but would likely still be a big upload. Alternatively do you have a way to reproduce this?

Casey
On February 26, 2016 at 10:47:25 AM, Nick Wolf (nickwolf7@gmail.com) wrote:

Enabled core dumps and following is the stacktrace.

#0  0x00007f205622ccc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56

#1  0x00007f20562300d8 in __GI_abort () at abort.c:89

#2  0x00007f2055b9e9c5 in os::abort(bool) () from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#3  0x00007f2055d1f607 in VMError::report_and_die() () from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#4  0x00007f2055d1fb8e in crash_handler(int, siginfo*, void*) ()

   from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#5  <signal handler called>

#6  0x00007f2055b953d1 in os::is_first_C_frame(frame*) () from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#7  0x00007f2055d1dcfd in VMError::report(outputStream*) ()

   from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#8  0x00007f2055d1f20a in VMError::report_and_die() () from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#9  0x00007f2055ba38af in JVM_handle_linux_signal () from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so

#10 <signal handler called>

#11 0x00000000007a453b in impala::ExprContext::GetValue(impala::Expr*, impala::TupleRow*) ()

#12 0x0000000000bcf45b in impala::UnionNode::EvalAndMaterializeExprs(std::vector<impala::ExprContext*, std::allocator<imp  ala::ExprContext*> > const&, bool, impala::Tuple**, impala::RowBatch*) ()

#13 0x0000000000bcf94c in impala::UnionNode::GetNext(impala::RuntimeState*, impala::RowBatch*, bool*) ()

#14 0x0000000000bb1ed7 in impala::PartitionedAggregationNode::Open(impala::RuntimeState*) ()

#15 0x0000000000b3f7ba in impala::PlanFragmentExecutor::OpenInternal() ()

#16 0x0000000000b3fecd in impala::PlanFragmentExecutor::Open() ()

#17 0x00000000009c7d3a in impala::FragmentMgr::FragmentExecState::Exec() ()

#18 0x00000000009c32dd in impala::FragmentMgr::FragmentExecThread(impala::FragmentMgr::FragmentExecState*) ()

#19 0x0000000000a482bd in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()  >, impala::Promise<long>*) ()

#20 0x0000000000a49164 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string co  nst&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::va  lue<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run()

    ()

#21 0x0000000000c49be3 in ?? ()

#22 0x00007f2058006182 in start_thread (arg=0x7f1f3023c700) at pthread_create.c:312

#23 0x00007f20562f047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

On Wed, Feb 24, 2016 at 4:44 PM, Nick Wolf <ni...@gmail.com> wrote:

KUDU Version 0.6.0

Two nodes of my 6 node cluster crashed with attached thread dump. This behavior is random and not sure what is causing the crash. 

Has anyone experienced this type of error?