You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Ganesh Murthy (Jira)" <ji...@apache.org> on 2021/03/10 15:43:00 UTC
[jira] [Commented] (DISPATCH-1999) [tcp] Occasional Router hang
when testing http over tcp using the "hey" tool
[ https://issues.apache.org/jira/browse/DISPATCH-1999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298914#comment-17298914 ]
Ganesh Murthy commented on DISPATCH-1999:
-----------------------------------------
Was able to get another pstack for the hung process
{noformat}
[gmurthy@localhost build]$ pstack 3837753
Thread 3 (Thread 0x7fbc506b8640 (LWP 3837755) "qdrouterd"):
#0 0x00007fbc627e5fdf in execute_cfa_program () from /lib64/libgcc_s.so.1
#1 0x00007fbc627e6ab8 in uw_frame_state_for () from /lib64/libgcc_s.so.1
#2 0x00007fbc627e88ab in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
#3 0x00007fbc62c2e8e6 in backtrace () from /lib64/libc.so.6
#4 0x00007fbc64a57601 in __interceptor_backtrace.part.0 () from /lib64/libasan.so.6
#5 0x00007fbc64071d35 in qd_alloc (desc=0x7fbc64840900 <__desc_qdr_action_t>, tpool=0x7fbc506b8548) at /home/gmurthy/opensource/qpid-dispatch/src/alloc_pool.c:345
#6 0x00007fbc64209f27 in new_qdr_action_t () at /home/gmurthy/opensource/qpid-dispatch/src/router_core/router_core_thread.c:33
#7 0x00007fbc641f5e4f in qdr_action (action_handler=0x7fbc641aaedf <qdr_delivery_continue_CT>, label=0x7fbc6439bee0 "delivery_continue") at /home/gmurthy/opensource/qpid-dispatch/src/router_core/router_core.c:383
#8 0x00007fbc6419eb84 in qdr_delivery_continue (core=0x61a000048080, in_dlv=0x616000176e10, settled=false) at /home/gmurthy/opensource/qpid-dispatch/src/router_core/delivery.c:212
#9 0x00007fbc64054f49 in handle_incoming_impl (conn=0x616000194810, close_pending=true) at /home/gmurthy/opensource/qpid-dispatch/src/adaptors/tcp_adaptor.c:242
#10 0x00007fbc6405b61b in handle_connection_event (e=0x60600010b0f0, qd_server=0x60f0000008b0, context=0x616000194810) at /home/gmurthy/opensource/qpid-dispatch/src/adaptors/tcp_adaptor.c:631
#11 0x00007fbc6429c3e6 in handle_event_with_context (e=0x60600010b0f0, qd_server=0x60f0000008b0, context=0x616000194810) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:804
#12 0x00007fbc6429c427 in do_handle_raw_connection_event (e=0x60600010b0f0, qd_server=0x60f0000008b0) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:810
#13 0x00007fbc642a0c76 in handle (qd_server=0x60f0000008b0, e=0x60600010b0f0, pn_conn=0x0, ctx=0x0) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:1090
#14 0x00007fbc642a0f37 in thread_run (arg=0x60f0000008b0) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:1122
#15 0x00007fbc6412a6ee in _thread_init (arg=0x60300003aff0) at /home/gmurthy/opensource/qpid-dispatch/src/posix/threading.c:172
#16 0x00007fbc63a413f9 in start_thread () from /lib64/libpthread.so.0
#17 0x00007fbc62c21903 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbc50eb9640 (LWP 3837754) "qdrouterd"):
#0 0x00007fbc63a44e99 in __pthread_mutex_unlock_usercnt () from /lib64/libpthread.so.0
#1 0x00007fbc64129f7e in sys_mutex_unlock (mutex=0x60a000000e00) at /home/gmurthy/opensource/qpid-dispatch/src/posix/threading.c:64
#2 0x00007fbc6420e3d5 in router_core_thread (arg=0x61a000048080) at /home/gmurthy/opensource/qpid-dispatch/src/router_core/router_core_thread.c:229
#3 0x00007fbc6412a6ee in _thread_init (arg=0x60300003acc0) at /home/gmurthy/opensource/qpid-dispatch/src/posix/threading.c:172
#4 0x00007fbc63a413f9 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fbc62c21903 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbc6218c380 (LWP 3837753) "qdrouterd"):
#0 0x00007fbc63a429d7 in __pthread_clockjoin_ex () from /lib64/libpthread.so.0
#1 0x00007fbc6412a95f in sys_thread_join (thread=0x60300003aff0) at /home/gmurthy/opensource/qpid-dispatch/src/posix/threading.c:202
#2 0x00007fbc642a82d3 in qd_server_run (qd=0x60c000000040) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:1486
#3 0x00000000004026f9 in main_process (config_path=0x7fff7c4c3635 "http-over-tcp.conf", python_pkgdir=0x405820 "/home/gmurthy/opensource/qpid-dispatch/install/lib/qpid-dispatch/python", test_hooks=false, fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:113
#4 0x0000000000404579 in main (argc=3, argv=0x7fff7c4c18a8) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:367
[gmurthy@localhost build]$
{noformat}
> [tcp] Occasional Router hang when testing http over tcp using the "hey" tool
> -----------------------------------------------------------------------------
>
> Key: DISPATCH-1999
> URL: https://issues.apache.org/jira/browse/DISPATCH-1999
> Project: Qpid Dispatch
> Issue Type: Bug
> Components: Protocol Adaptors
> Affects Versions: 1.15.0
> Reporter: Ganesh Murthy
> Priority: Major
> Attachments: http-over-tcp.conf
>
>
> To reproduce -
> # Start the router with the attached config
> # run the hey tool - ./hey_linux_amd64 -c 5 -z 10s [http://127.0.0.1:9000/index.html]
> # Wait for the hey test to finish. It might take about about 30 to 60 seconds to finish.
> # Press Ctrl + C to terminate the router
> # The router does not terminate. The output of pstack is as follows
> {noformat}
> [gmurthy@localhost build]$ pstack 3833922
> Thread 2 (Thread 0x7f308cc01640 (LWP 3833932) "qdrouterd"):
> #0 0x00007f309f7936c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00007f309fe78240 in sys_cond_wait (cond=0x60a000000d40, held_mutex=0x60a000000e00) at /home/gmurthy/opensource/qpid-dispatch/src/posix/threading.c:92
> #2 0x00007f309ff5cb0b in router_core_thread (arg=0x61a000048080) at /home/gmurthy/opensource/qpid-dispatch/src/router_core/router_core_thread.c:221
> #3 0x00007f309fe786ee in _thread_init (arg=0x60300003acc0) at /home/gmurthy/opensource/qpid-dispatch/src/posix/threading.c:172
> #4 0x00007f309f78d3f9 in start_thread () from /lib64/libpthread.so.0
> #5 0x00007f309e96d903 in clone () from /lib64/libc.so.6
> Thread 1 (Thread 0x7f309ded8380 (LWP 3833922) "qdrouterd"):
> #0 0x00007f309fda96c2 in handle_connection_event (e=0x60600014e710, qd_server=0x60f0000008b0, context=0x616000112c10) at /home/gmurthy/opensource/qpid-dispatch/src/adaptors/tcp_adaptor.c:633
> #1 0x00007f309ffeafd4 in handle_event_with_context (e=0x60600014e710, qd_server=0x60f0000008b0, context=0x616000112c10) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:804
> #2 0x00007f309ffeb015 in do_handle_raw_connection_event (e=0x60600014e710, qd_server=0x60f0000008b0) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:810
> #3 0x00007f309ffef864 in handle (qd_server=0x60f0000008b0, e=0x60600014e710, pn_conn=0x0, ctx=0x0) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:1090
> #4 0x00007f309ffefb25 in thread_run (arg=0x60f0000008b0) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:1122
> #5 0x00007f309fff6e0d in qd_server_run (qd=0x60c000000040) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:1484
> #6 0x00000000004026f9 in main_process (config_path=0x7ffe7d076635 "http-over-tcp.conf", python_pkgdir=0x405820 "/home/gmurthy/opensource/qpid-dispatch/install/lib/qpid-dispatch/python", test_hooks=false, fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:113
> #7 0x0000000000404579 in main (argc=3, argv=0x7ffe7d075228) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:367
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org