You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Jiri Daněk (Jira)" <ji...@apache.org> on 2021/04/02 11:32:00 UTC
[jira] [Updated] (DISPATCH-385) Router crashes when on-demand host
is no longer reachable
[ https://issues.apache.org/jira/browse/DISPATCH-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jiri Daněk updated DISPATCH-385:
--------------------------------
Description:
Router will crash if the host for an on-demand connector is unreachable.
To reproduce:
start the 6 router configs in qpid-dispatch/tests/config-6
start a broker on a different box (I used a qpidd on port 11000 on mrg30)
create an on-demand connector to the broker on one of the routers (I use Y)
verify a connection to the broker has been automatically created and is open
kill/block the communications to the broker (I stop my vpn, but there may be better ways)
after a few minutes, the router that the broker was connected to will crash
here is the bt:
{noformat}
#0 0x00007ffff6b9ea28 in raise () from /lib64/libc.so.6
#1 0x00007ffff6ba062a in abort () from /lib64/libc.so.6
#2 0x00007ffff6b97227 in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007ffff6b972d2 in __assert_fail () from /lib64/libc.so.6
#4 0x00007ffff7b9face in sys_mutex_lock (mutex=0x7fffd4003e00)
at /home/eallen/workspace/qpid-dispatch/src/posix/threading.c:71
#5 0x00007ffff7bac174 in qdr_forward_deliver_CT (core=0x9264d0,
link=0x7fffe403a4c0, dlv=0x7fffe4072ec0)
at /home/eallen/workspace/qpid-dispatch/src/router_core/forwarder.c:132
#6 0x00007ffff7bacfeb in qdr_forward_closest_CT (core=0x9264d0,
addr=0x7fffe4024fc0, msg=0x7fffe4060e50, in_delivery=0x0,
exclude_inprocess=true, control=false)
at /home/eallen/workspace/qpid-dispatch/src/router_core/forwarder.c:405
#7 0x00007ffff7badd3a in qdr_forward_message_CT (core=0x9264d0,
addr=0x7fffe4024fc0, msg=0x7fffe4060e50, in_delivery=0x0,
exclude_inprocess=true, control=false)
at /home/eallen/workspace/qpid-dispatch/src/router_core/forwarder.c:707
#8 0x00007ffff7bb73f6 in qdr_send_to_CT (core=0x9264d0, action=0x9749d0,
discard=false)
at /home/eallen/workspace/qpid-dispatch/src/router_core/transfer.c:581
#9 0x00007ffff7bb18ea in router_core_thread (arg=0x9264d0)
at /home/eallen/workspace/qpid-dispatch/src/router_core/router_core_thread.c:71
#10 0x00007ffff770d60a in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff6c6c59d in clone () from /lib64/libc.so.6
{noformat}
I'm using master qpid-proton 0.14.0-SNAPSHOT
master qpid-dispatch 0.7.0
Here is the routers debug output after the vpn was killed and just before the crash
{noformat}
Mon Jun 13 11:34:14 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:34:23 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:34:34 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:34:44 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:35:10 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.D link removed
Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.C link removed
Mon Jun 13 11:35:10 2016 ROUTER (trace) Entered Router Flux Mode
Mon Jun 13 11:35:10 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.D link set: link_id=0
Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.C link set: link_id=1
qdrouterd: /home/eallen/workspace/qpid-dispatch/src/posix/threading.c:71: sys_mutex_lock: Assertion `result == 0' failed.
Program received signal SIGABRT, Aborted.
{noformat}
was:
Router will crash if the host for an on-demand connector is unreachable.
To reproduce:
start the 6 router configs in qpid-dispatch/tests/config-6
start a broker on a different box (I used a qpidd on port 11000 on mrg30)
create an on-demand connector to the broker on one of the routers (I use Y)
verify a connection to the broker has been automatically created and is open
kill/block the communications to the broker (I stop my vpn, but there may be better ways)
after a few minutes, the router that the broker was connected to will crash
here is the bt:
#0 0x00007ffff6b9ea28 in raise () from /lib64/libc.so.6
#1 0x00007ffff6ba062a in abort () from /lib64/libc.so.6
#2 0x00007ffff6b97227 in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007ffff6b972d2 in __assert_fail () from /lib64/libc.so.6
#4 0x00007ffff7b9face in sys_mutex_lock (mutex=0x7fffd4003e00)
at /home/eallen/workspace/qpid-dispatch/src/posix/threading.c:71
#5 0x00007ffff7bac174 in qdr_forward_deliver_CT (core=0x9264d0,
link=0x7fffe403a4c0, dlv=0x7fffe4072ec0)
at /home/eallen/workspace/qpid-dispatch/src/router_core/forwarder.c:132
#6 0x00007ffff7bacfeb in qdr_forward_closest_CT (core=0x9264d0,
addr=0x7fffe4024fc0, msg=0x7fffe4060e50, in_delivery=0x0,
exclude_inprocess=true, control=false)
at /home/eallen/workspace/qpid-dispatch/src/router_core/forwarder.c:405
#7 0x00007ffff7badd3a in qdr_forward_message_CT (core=0x9264d0,
addr=0x7fffe4024fc0, msg=0x7fffe4060e50, in_delivery=0x0,
exclude_inprocess=true, control=false)
at /home/eallen/workspace/qpid-dispatch/src/router_core/forwarder.c:707
#8 0x00007ffff7bb73f6 in qdr_send_to_CT (core=0x9264d0, action=0x9749d0,
discard=false)
at /home/eallen/workspace/qpid-dispatch/src/router_core/transfer.c:581
#9 0x00007ffff7bb18ea in router_core_thread (arg=0x9264d0)
at /home/eallen/workspace/qpid-dispatch/src/router_core/router_core_thread.c:71
#10 0x00007ffff770d60a in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff6c6c59d in clone () from /lib64/libc.so.6
I'm using master qpid-proton 0.14.0-SNAPSHOT
master qpid-dispatch 0.7.0
Here is the routers debug output after the vpn was killed and just before the crash
Mon Jun 13 11:34:14 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:34:23 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:34:34 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:34:44 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:35:10 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.D link removed
Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.C link removed
Mon Jun 13 11:35:10 2016 ROUTER (trace) Entered Router Flux Mode
Mon Jun 13 11:35:10 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.D link set: link_id=0
Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.C link set: link_id=1
qdrouterd: /home/eallen/workspace/qpid-dispatch/src/posix/threading.c:71: sys_mutex_lock: Assertion `result == 0' failed.
Program received signal SIGABRT, Aborted.
> Router crashes when on-demand host is no longer reachable
> ---------------------------------------------------------
>
> Key: DISPATCH-385
> URL: https://issues.apache.org/jira/browse/DISPATCH-385
> Project: Qpid Dispatch
> Issue Type: Bug
> Components: Router Node
> Affects Versions: 0.7.0
> Reporter: Ernest Allen
> Assignee: Ganesh Murthy
> Priority: Major
> Fix For: Backlog
>
> Attachments: A.conf, B.conf, C.conf, D.conf, X.conf, Y.conf
>
>
> Router will crash if the host for an on-demand connector is unreachable.
> To reproduce:
> start the 6 router configs in qpid-dispatch/tests/config-6
> start a broker on a different box (I used a qpidd on port 11000 on mrg30)
> create an on-demand connector to the broker on one of the routers (I use Y)
> verify a connection to the broker has been automatically created and is open
> kill/block the communications to the broker (I stop my vpn, but there may be better ways)
> after a few minutes, the router that the broker was connected to will crash
> here is the bt:
> {noformat}
> #0 0x00007ffff6b9ea28 in raise () from /lib64/libc.so.6
> #1 0x00007ffff6ba062a in abort () from /lib64/libc.so.6
> #2 0x00007ffff6b97227 in __assert_fail_base () from /lib64/libc.so.6
> #3 0x00007ffff6b972d2 in __assert_fail () from /lib64/libc.so.6
> #4 0x00007ffff7b9face in sys_mutex_lock (mutex=0x7fffd4003e00)
> at /home/eallen/workspace/qpid-dispatch/src/posix/threading.c:71
> #5 0x00007ffff7bac174 in qdr_forward_deliver_CT (core=0x9264d0,
> link=0x7fffe403a4c0, dlv=0x7fffe4072ec0)
> at /home/eallen/workspace/qpid-dispatch/src/router_core/forwarder.c:132
> #6 0x00007ffff7bacfeb in qdr_forward_closest_CT (core=0x9264d0,
> addr=0x7fffe4024fc0, msg=0x7fffe4060e50, in_delivery=0x0,
> exclude_inprocess=true, control=false)
> at /home/eallen/workspace/qpid-dispatch/src/router_core/forwarder.c:405
> #7 0x00007ffff7badd3a in qdr_forward_message_CT (core=0x9264d0,
> addr=0x7fffe4024fc0, msg=0x7fffe4060e50, in_delivery=0x0,
> exclude_inprocess=true, control=false)
> at /home/eallen/workspace/qpid-dispatch/src/router_core/forwarder.c:707
> #8 0x00007ffff7bb73f6 in qdr_send_to_CT (core=0x9264d0, action=0x9749d0,
> discard=false)
> at /home/eallen/workspace/qpid-dispatch/src/router_core/transfer.c:581
> #9 0x00007ffff7bb18ea in router_core_thread (arg=0x9264d0)
> at /home/eallen/workspace/qpid-dispatch/src/router_core/router_core_thread.c:71
> #10 0x00007ffff770d60a in start_thread () from /lib64/libpthread.so.0
> #11 0x00007ffff6c6c59d in clone () from /lib64/libc.so.6
> {noformat}
> I'm using master qpid-proton 0.14.0-SNAPSHOT
> master qpid-dispatch 0.7.0
> Here is the routers debug output after the vpn was killed and just before the crash
> {noformat}
> Mon Jun 13 11:34:14 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
> Mon Jun 13 11:34:23 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
> Mon Jun 13 11:34:34 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
> Mon Jun 13 11:34:44 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
> Mon Jun 13 11:35:10 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
> Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.D link removed
> Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.C link removed
> Mon Jun 13 11:35:10 2016 ROUTER (trace) Entered Router Flux Mode
> Mon Jun 13 11:35:10 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
> Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.D link set: link_id=0
> Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.C link set: link_id=1
> qdrouterd: /home/eallen/workspace/qpid-dispatch/src/posix/threading.c:71: sys_mutex_lock: Assertion `result == 0' failed.
> Program received signal SIGABRT, Aborted.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org