You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Jiri Daněk (Jira)" <ji...@apache.org> on 2020/08/07 06:36:00 UTC
[jira] [Commented] (PROTON-2259) Threadercizer test causes
intermittent CI build failures
[ https://issues.apache.org/jira/browse/PROTON-2259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172929#comment-17172929 ]
Jiri Daněk commented on PROTON-2259:
------------------------------------
Investigation comment by [~chug], originally posted on PROTON-2230
{quote}
I put in a bunch of print statements to see what's what. Several things:
* ctest passes no command line args to c-threaderciser. The reason it prints time=1 is because that's the default. Same with threads=8.
* Running under valgrind/memcheck it takes the 57 seconds when it passes. It's that slow.
* When it fails: the main thread has exited millisleep(runtime*1000) and is sitting in the user_threads pthread_join loop where the threads apparently never join.
* The other threads are wailing away having a grand old time
Had to use valgrind gdb server to get a useful backtrace on the threads.
{noformat}
Thread 9 (Thread 12699):
#0 0x0000000004a01b0d in recvmsg () from /lib64/libc.so.6
#1 0x0000000004a1d48e in __check_pf () from /lib64/libc.so.6
#2 0x00000000049eb19f in getaddrinfo () from /lib64/libc.so.6
#3 0x000000000484f31a in pgetaddrinfo (host=0x514313a "0.0.0.0", port=0x5143142 "41675", flags=0, res=0x5143100) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:1544
#4 0x000000000484f544 in pn_proactor_connect2 (p=0x5120590, c=0x5142460, t=0x0, addr=0x512c190 "0.0.0.0:41675") at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:1581
#5 0x0000000004853ef6 in pn_proactor_connect (p=0x5120590, c=0x5142460, addr=0x512c190 "0.0.0.0:41675") at /home/chug/git/qpid-proton/c/src/proactor/proactor-internal.c:90
#6 0x0000000000402b38 in cpool_connect (cp=0x1ffefff078, proactor=0x5120590, addr=0x512c190 "0.0.0.0:41675") at /home/chug/git/qpid-proton/c/tests/threaderciser.c:229
#7 0x0000000000403508 in handle (g=0x1ffefff020, e=0x5142210) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:425
#8 0x0000000000403661 in proactor_thread (void_g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:476
#9 0x00000000048e64e2 in start_thread () from /lib64/libpthread.so.0
#10 0x0000000004a006c3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 12698):
#0 0x000000000401d4c8 in __open_nocancel () from /lib64/ld-linux-x86-64.so.2
#1 0x0000000004011c4a in _dl_sysdep_read_whole_file () from /lib64/ld-linux-x86-64.so.2
#2 0x0000000004018538 in _dl_load_cache_lookup () from /lib64/ld-linux-x86-64.so.2
#3 0x000000000400a69a in _dl_map_object () from /lib64/ld-linux-x86-64.so.2
#4 0x0000000004014d55 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#5 0x0000000004a3b4f8 in _dl_catch_exception () from /lib64/libc.so.6
#6 0x00000000040148fe in _dl_open () from /lib64/ld-linux-x86-64.so.2
#7 0x0000000004a3a9c1 in do_dlopen () from /lib64/libc.so.6
#8 0x0000000004a3b4f8 in _dl_catch_exception () from /lib64/libc.so.6
#9 0x0000000004a3b5c3 in _dl_catch_error () from /lib64/libc.so.6
#10 0x0000000004a3aac7 in dlerror_run () from /lib64/libc.so.6
#11 0x0000000004a3ab5a in __libc_dlopen_mode () from /lib64/libc.so.6
#12 0x0000000004a22384 in nss_load_library () from /lib64/libc.so.6
#13 0x0000000004a22c79 in __nss_lookup_function () from /lib64/libc.so.6
#14 0x0000000004a22e10 in __nss_lookup () from /lib64/libc.so.6
#15 0x0000000004a148db in getservbyname_r@@GLIBC_2.2.5 () from /lib64/libc.so.6
#16 0x00000000049e8338 in gaih_inet_serv.isra () from /lib64/libc.so.6
#17 0x00000000049e8e06 in gaih_inet.constprop () from /lib64/libc.so.6
#18 0x00000000049ea8a1 in getaddrinfo () from /lib64/libc.so.6
#19 0x000000000484f31a in pgetaddrinfo (host=0x51371aa "invalid", port=0x51371b2 "address", flags=0, res=0x5137170) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:1544
#20 0x000000000484f544 in pn_proactor_connect2 (p=0x5120590, c=0x51364d0, t=0x0, addr=0x8d25970 "invalid:address") at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:1581
#21 0x0000000004853ef6 in pn_proactor_connect (p=0x5120590, c=0x51364d0, addr=0x8d25970 "invalid:address") at /home/chug/git/qpid-proton/c/src/proactor/proactor-internal.c:90
#22 0x0000000000402b38 in cpool_connect (cp=0x1ffefff078, proactor=0x5120590, addr=0x8d25970 "invalid:address") at /home/chug/git/qpid-proton/c/tests/threaderciser.c:229
#23 0x0000000000403135 in global_connect (g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:367
#24 0x00000000004031ba in global_do_stuff (g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:378
#25 0x0000000000403345 in user_thread (void_g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:397
#26 0x00000000048e64e2 in start_thread () from /lib64/libpthread.so.0
#27 0x0000000004a006c3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 12696):
#0 0x0000000004985e0b in __lll_lock_wait_private () from /lib64/libc.so.6
#1 0x0000000004a1d6e9 in __check_pf () from /lib64/libc.so.6
#2 0x00000000049eb19f in getaddrinfo () from /lib64/libc.so.6
#3 0x000000000484f31a in pgetaddrinfo (host=0x0, port=0x51352c1 "0", flags=17, res=0x7d23908) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:1544
#4 0x000000000484fa0a in pn_proactor_listen (p=0x5120590, l=0x51351f0, addr=0x7d23970 ":0", backlog=16) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:1679
#5 0x0000000000402dd7 in lpool_listen (lp=0x1ffefff030, proactor=0x5120590) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:285
#6 0x00000000004031e9 in global_do_stuff (g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:379
#7 0x0000000000403345 in user_thread (void_g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:397
#8 0x00000000048e64e2 in start_thread () from /lib64/libpthread.so.0
#9 0x0000000004a006c3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 12695):
#0 0x00000000048ecd45 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000484c5bf in suspend (p=0x5120590, ts=0x512afe0) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:518
#2 0x0000000004852efc in proactor_do_epoll (p=0x5120590, can_block=true) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:2761
#3 0x0000000004853336 in pn_proactor_wait (p=0x5120590) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:2823
#4 0x0000000000403642 in proactor_thread (void_g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:473
#5 0x00000000048e64e2 in start_thread () from /lib64/libpthread.so.0
#6 0x0000000004a006c3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 12694):
#0 0x0000000004985e0b in __lll_lock_wait_private () from /lib64/libc.so.6
#1 0x0000000004a1d6e9 in __check_pf () from /lib64/libc.so.6
#2 0x00000000049eb19f in getaddrinfo () from /lib64/libc.so.6
#3 0x000000000484f31a in pgetaddrinfo (host=0x0, port=0x512b7b1 "0", flags=17, res=0x6d21908) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:1544
#4 0x000000000484fa0a in pn_proactor_listen (p=0x5120590, l=0x512b6e0, addr=0x6d21970 ":0", backlog=16) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:1679
#5 0x0000000000402dd7 in lpool_listen (lp=0x1ffefff030, proactor=0x5120590) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:285
#6 0x00000000004031e9 in global_do_stuff (g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:379
#7 0x0000000000403345 in user_thread (void_g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:397
#8 0x00000000048e64e2 in start_thread () from /lib64/libpthread.so.0
#9 0x0000000004a006c3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 12693):
#0 0x00000000048ecd45 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000484c5bf in suspend (p=0x5120590, ts=0x5123250) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:518
#2 0x0000000004852efc in proactor_do_epoll (p=0x5120590, can_block=true) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:2761
#3 0x0000000004853336 in pn_proactor_wait (p=0x5120590) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:2823
#4 0x0000000000403642 in proactor_thread (void_g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:473
#5 0x00000000048e64e2 in start_thread () from /lib64/libpthread.so.0
#6 0x0000000004a006c3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 12692):
#0 0x00000000048f0a4b in connect () from /lib64/libpthread.so.0
#1 0x000000000484f1f8 in pconnection_maybe_connect_lh (pc=0x512db50) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:1518
#2 0x000000000484f582 in pn_proactor_connect2 (p=0x5120590, c=0x512d120, t=0x0, addr=0x5d1f970 "0.0.0.0:36609") at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:1585
#3 0x0000000004853ef6 in pn_proactor_connect (p=0x5120590, c=0x512d120, addr=0x5d1f970 "0.0.0.0:36609") at /home/chug/git/qpid-proton/c/src/proactor/proactor-internal.c:90
#4 0x0000000000402b38 in cpool_connect (cp=0x1ffefff078, proactor=0x5120590, addr=0x5d1f970 "0.0.0.0:36609") at /home/chug/git/qpid-proton/c/tests/threaderciser.c:229
#5 0x0000000000403135 in global_connect (g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:367
#6 0x00000000004031ba in global_do_stuff (g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:378
#7 0x0000000000403345 in user_thread (void_g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:397
#8 0x00000000048e64e2 in start_thread () from /lib64/libpthread.so.0
#9 0x0000000004a006c3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 12697):
#0 0x0000000004869f8f in pn_class_decref (clazz=0x48a7520 <list_clazz>, object=0x5121cc0) at /home/chug/git/qpid-proton/c/src/core/object/object.c:95
#1 0x000000000486a531 in pn_decref (object=0x5121cc0) at /home/chug/git/qpid-proton/c/src/core/object/object.c:256
#2 0x0000000004880000 in pn_event_finalize (event=0x5122a90) at /home/chug/git/qpid-proton/c/src/core/event.c:243
#3 0x000000000488017e in pn_event_finalize_cast (object=0x5122a90) at /home/chug/git/qpid-proton/c/src/core/event.c:271
#4 0x0000000004869fb6 in pn_class_decref (clazz=0x48a7c20 <pn_event.class&amp;gt;, object=0x5122a90) at /home/chug/git/qpid-proton/c/src/core/object/object.c:98
#5 0x000000000486a531 in pn_decref (object=0x5122a90) at /home/chug/git/qpid-proton/c/src/core/object/object.c:256
#6 0x000000000487fe3b in pn_collector_next (collector=0x5121c40) at /home/chug/git/qpid-proton/c/src/core/event.c:197
#7 0x00000000048505d5 in listener_batch_next (batch=0x5121b90) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:1924
#8 0x0000000004853ec4 in pn_event_batch_next (batch=0x5121b90) at /home/chug/git/qpid-proton/c/src/proactor/proactor-internal.c:84
#9 0x000000000040368a in proactor_thread (void_g=0x1ffefff020) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:475
#10 0x00000000048e64e2 in start_thread () from /lib64/libpthread.so.0
#11 0x0000000004a006c3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 12598):
#0 0x00000000048e7b27 in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
#1 0x0000000000403e15 in main (argc=1, argv=0x1ffefff288) at /home/chug/git/qpid-proton/c/tests/threaderciser.c:596
(gdb)
{noformat}
{quote}
> Threadercizer test causes intermittent CI build failures
> --------------------------------------------------------
>
> Key: PROTON-2259
> URL: https://issues.apache.org/jira/browse/PROTON-2259
> Project: Qpid Proton
> Issue Type: Bug
> Components: proton-c
> Affects Versions: proton-c-0.31.0
> Reporter: Andrew Stitcher
> Priority: Major
>
> The threadercizer test run fails - usually locks up - about 1 in 10 of the various builds. There seems to be no particular pattern, failing on occasionally on all the builds (except Windows, but I'm not sure it runs there anyway!).
> Given that we run a large munber of CI jobs on various platforms (10-20) this means that there is very often a failure due to this tst.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org