You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tim Armstrong (Jira)" <ji...@apache.org> on 2020/03/13 16:28:00 UTC
[jira] [Resolved] (IMPALA-6073) No response from
expr-codegen-test/expr-test
[ https://issues.apache.org/jira/browse/IMPALA-6073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-6073.
-----------------------------------
Resolution: Cannot Reproduce
> No response from expr-codegen-test/expr-test
> --------------------------------------------
>
> Key: IMPALA-6073
> URL: https://issues.apache.org/jira/browse/IMPALA-6073
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.9.0
> Environment: ubuntu 14.04.5 LTS
> Reporter: Jin Chul Kim
> Assignee: Philip Martin
> Priority: Major
>
> I checked out HEAD of Impala repo. There is no any code modification on my local repo.
> After full build, ./build/debug/exprs/expr-test and ./build/debug/exprs/expr-codegen-test seem to be hung. The other gtests in be are working appropriately. Some threads are waiting and the other threads are on sleep.
> I guess this issue is similar to the deadlock: https://issues.apache.org/jira/browse/HDFS-11851
> Please let me know if you have any workaround.
> Here are stack traces on expr-codegen-test using GDB:
> {code:java}
> jinchulkim@ubuntu:~/workspace/Impala/be$ gdb ./build/debug/exprs/expr-codegen-test
> ...
> (gdb) info thread
> Id Target Id Frame
> 16 Thread 0x7fffe20ae700 (LWP 23374) "expr-codegen-te" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
> 15 Thread 0x7fffe21af700 (LWP 23373) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> 14 Thread 0x7fffe22b0700 (LWP 23372) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> 13 Thread 0x7fffe23b1700 (LWP 23371) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> 12 Thread 0x7fffe24b2700 (LWP 23370) "expr-codegen-te" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
> 11 Thread 0x7fffe2c95700 (LWP 23369) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> 10 Thread 0x7fffe2d96700 (LWP 23368) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> 9 Thread 0x7fffe2e97700 (LWP 23367) "expr-codegen-te" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
> 8 Thread 0x7fffe915a700 (LWP 23366) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> 7 Thread 0x7fffe925b700 (LWP 23365) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> 6 Thread 0x7fffe935c700 (LWP 23364) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> 5 Thread 0x7fffe945d700 (LWP 23363) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> 4 Thread 0x7fffee536700 (LWP 23362) "expr-codegen-te" 0x00007ffff037bb9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
> 3 Thread 0x7fffeed37700 (LWP 23361) "expr-codegen-te" 0x00007ffff037bb9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
> 2 Thread 0x7fffef538700 (LWP 23356) "expr-codegen-te" 0x00007ffff0067dfd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
> * 1 Thread 0x7fffef5408c0 (LWP 23150) "expr-codegen-te" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> (gdb) where
> #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> #1 0x00007ffff0376649 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
> #2 0x00007ffff0376470 in __GI___pthread_mutex_lock (mutex=0x7ffff4080600 <jvmMutex>) at ../nptl/pthread_mutex_lock.c:79
> #3 0x00007ffff3e7b666 in mutexLock (m=<optimized out>) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28
> #4 0x00007ffff3e73a11 in setTLSExceptionStrings (rootCause=0x0, stackTrace=0x0) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581
> #5 0x00007ffff3e73393 in printExceptionAndFreeV (env=0x307c1d8, exc=0x301adc0, noPrintFlags=<optimized out>, fmt=0x7ffff3e7bf6e "loadFileSystems", ap=0x7fffffffaab0) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183
> #6 0x00007ffff3e735ef in printExceptionAndFree (env=<optimized out>, exc=<optimized out>, noPrintFlags=<optimized out>, fmt=<optimized out>) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213
> #7 0x00007ffff3e74880 in getGlobalJNIEnv () at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463
> #8 getJNIEnv () at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:528
> #9 0x00007ffff7845a1c in impala::JniUtil::Init () at /home/jinchulkim/workspace/Impala/be/src/util/jni-util.cc:105
> #10 0x00007ffff7e3c221 in impala::InitCommonRuntime (argc=1, argv=0x7fffffffb1e8, init_jvm=true, test_mode=impala::TestInfo::BE_TEST) at /home/jinchulkim/workspace/Impala/be/src/common/init.cc:236
> #11 0x00000000008333fc in main (argc=1, argv=0x7fffffffb1e8) at /home/jinchulkim/workspace/Impala/be/src/exprs/expr-codegen-test.cc:361
> (gdb) thr 2
> [Switching to thread 2 (Thread 0x7fffef538700 (LWP 23356))]
> #0 0x00007ffff0067dfd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
> 81 ../sysdeps/unix/syscall-template.S: No such file or directory.
> (gdb) where
> #0 0x00007ffff0067dfd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
> #1 0x00007ffff0067c94 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:137
> #2 0x00007ffff7e3b304 in LogMaintenanceThread () at /home/jinchulkim/workspace/Impala/be/src/common/init.cc:113
> #3 0x00007ffff56e4a8e in boost::detail::function::void_function_invoker0<void (*)(), void>::invoke (function_ptr=...) at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:112
> #4 0x00007ffff7b22f22 in boost::function0<void>::operator() (this=0x7fffef537ca0) at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #5 0x00007ffff78c0ba3 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name=..., category=..., functor=..., thread_started=0x7fffffffaa60) at /home/jinchulkim/workspace/Impala/be/src/util/thread.cc:352
> #6 0x00007ffff78ca32c in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0x2fb47c0, f=@0x2fb47b8: 0x7ffff78c0884 <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...)
> at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
> #7 0x00007ffff78ca26f in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0x2fb47b8)
> at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
> #8 0x00007ffff78ca232 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run()
> (this=0x2fb4600) at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
> #9 0x000000000090c53a in thread_proxy ()
> #10 0x00007ffff0374184 in start_thread (arg=0x7fffef538700) at pthread_create.c:312
> #11 0x00007ffff00a137d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
> (gdb)
> {code}
> Here are the relevant environment variables:
> {code:java}
> jinchulkim@ubuntu:~/workspace/Impala/be$ echo $JAVA_HOME
> /usr/lib/jvm/java-7-openjdk-amd64
> jinchulkim@ubuntu:~/workspace/Impala/be$ echo $BOOST_LIBRARYDIR
> /usr/lib/x86_64-linux-gnu
> jinchulkim@ubuntu:~/workspace/Impala/be$ echo $LD_LIBRARY_PATH
> :/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/home/jinchulkim/workspace/Impala/toolchain/cdh_components/hadoop-2.6.0-cdh5.14.0-SNAPSHOT//lib/native:/home/jinchulkim/workspace/Impala/toolchain/snappy-1.1.4/lib:/home/jinchulkim/workspace/Impala/../Impala-lzo/build:/home/jinchulkim/workspace/Impala/toolchain/gcc-4.9.2/lib64:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/home/jinchulkim/workspace/Impala/toolchain/cdh_components/hadoop-2.6.0-cdh5.14.0-SNAPSHOT//lib/native:/home/jinchulkim/workspace/Impala/toolchain/snappy-1.1.4/lib:/home/jinchulkim/workspace/Impala/../Impala-lzo/build:/home/jinchulkim/workspace/Impala/toolchain/gcc-4.9.2/lib64
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)