[jira] [Created] (ZOOKEEPER-2200) Perl ZooKeeper locks up during heavy load

["KARA VAN HORN (JIRA)" <ji...@apache.org>]

KARA VAN HORN created ZOOKEEPER-2200:
----------------------------------------

             Summary: Perl ZooKeeper locks up during heavy load
                 Key: ZOOKEEPER-2200
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2200
             Project: ZooKeeper
          Issue Type: Bug
          Components: c client
    Affects Versions: 3.4.6
         Environment: CentOS 5.8
            Reporter: KARA VAN HORN


We are using Perl Net::ZooKeeper (0.38) and Net::ZooKeeper::Lock (0.03) libraries.  Deadlock appears to occur at the end during lock cleanup activity.  Here is a stack dump (sensitive names changed):

Thread 2 (Thread 0x2ac6fbfa3940 (LWP 13292)):
#0  0x00002ac6f5aed654 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002ac6f5aeb47b in pthread_cond_signal@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#2  0x00002ac6f835539c in _zk_watcher (handle=<value optimized out>, type=2, state=3, path=<value optimized out>, context=0x33f3ce0) at ZooKeeper.xs:179
#3  0x00002ac6f856d942 in do_foreach_watcher (zh=0x33e4fb0, type=2, state=3, path=0x33f3f50 "/lock/cmts/cisco_device1.net-0001851215", list=0x33ed290)
    at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zk_hashtable.c:279
#4  deliverWatchers (zh=0x33e4fb0, type=2, state=3, path=0x33f3f50 "/lock/cmts/cisco_device1.net-0001851215", list=0x33ed290)
    at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zk_hashtable.c:321
#5  0x00002ac6f8564966 in process_completions (zh=0x33e4fb0) at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zookeeper.c:2114
#6  0x00002ac6f856e101 in do_completion (v=<value optimized out>) at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/mt_adaptor.c:466
#7  0x00002ac6f5ae683d in start_thread (arg=<value optimized out>) at pthread_create.c:301
#8  0x00002ac6f5dd1fcd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2ac6f6056af0 (LWP 12972)):
#0  0x00002ac6f5ae7c65 in pthread_join (threadid=47034119371072, thread_return=0x0) at pthread_join.c:89
#1  0x00002ac6f856e7de in adaptor_finish (zh=0x33e4fb0) at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/mt_adaptor.c:293
#2  0x00002ac6f8566cdc in zookeeper_close (zh=0x33e4fb0) at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zookeeper.c:2536
#3  0x00002ac6f8357222 in XS_Net__ZooKeeper_DESTROY (my_perl=0x20df010, cv=<value optimized out>) at ZooKeeper.xs:885
#4  0x00002ac6f4b38af6 in Perl_pp_entersub () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#5  0x00002ac6f4adb8d7 in ?? () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#6  0x00002ac6f4adf720 in Perl_call_sv () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#7  0x00002ac6f4b3d3c6 in Perl_sv_clear () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#8  0x00002ac6f4b3db70 in Perl_sv_free () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#9  0x00002ac6f4b6025c in Perl_free_tmps () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#10 0x00002ac6f4adf78a in Perl_call_sv () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#11 0x00002ac6f4b3d3c6 in Perl_sv_clear () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#12 0x00002ac6f4b3db70 in Perl_sv_free () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#13 0x00002ac6f4b3b0e5 in ?? () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#14 0x00002ac6f4b3b141 in Perl_sv_clean_objs () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#15 0x00002ac6f4ae185e in perl_destruct () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#16 0x0000000000401773 in main ()

There are about 4 out of 10,000 processes that end up in deadlock, and according to our web searches, the only reason pthread_cond_signal would lock is due to it waiting on an already destroyed condition.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)