You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "michael goulish (JIRA)" <ji...@apache.org> on 2019/04/02 18:32:00 UTC
[jira] [Commented] (DISPATCH-1309) Various crashes in 1.6 release
[ https://issues.apache.org/jira/browse/DISPATCH-1309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808007#comment-16808007 ]
michael goulish commented on DISPATCH-1309:
-------------------------------------------
OK! I thought Mercury might help reproduce this more easily, and ... it did.
I made a 13-router star-shaped network ( the Death Star ) – 12 routers in a circle and one at the center.
There was 1 receiver at every router on the circle all hoping for 1 million messages. 1 sender at the center router, trying to make all the receivers happy.
It ran for a good amount of time – I could see the traffic turning all the links green using the console – and then 7 routers crashed all at once, generating 5 different types of core files.
Which follow.
##############################################################
# Type 1
##############################################################
#0 0x00007f23bbbb0750 in raise () from /lib64/libc.so.6
#1 0x00007f23bbbb1d31 in abort () from /lib64/libc.so.6
#2 0x00007f23bbba905a in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007f23bbba90d2 in __assert_fail () from /lib64/libc.so.6
#4 0x00007f23bc9b8e6f in __pthread_tpp_change_priority () from /lib64/libpthread.so.0
#5 0x00007f23bc9af8fb in __pthread_mutex_lock_full () from /lib64/libpthread.so.0
#6 0x00007f23bd044309 in qdra_config_address_create_CT (core=0x7f23a805e0d8,
name=<optimized out>, query=0x7f23a00307d8, in_body=<optimized out>)
at /home/mick/latest/qpid-dispatch-1.6.0/src/router_core/agent_config_address.c:446
#7 0x0000000000000000 in ?? ()
in qdra_config_address_create_CT
(gdb) list
441 addr->priority = priority;
442 pattern = 0;
443
444 qd_iterator_reset_view(iter, ITER_VIEW_ALL);
445 qd_parse_tree_add_pattern(core->addr_parse_tree, iter, addr);
446 DEQ_INSERT_TAIL(core->addr_config, addr);
447
448 //
449 // Compose the result map for the response.
450 //
##############################################################
# Type 2
##############################################################
#0 connection_wake (conn=<optimized out>)
at /home/mick/latest/qpid-dispatch-1.6.0/src/remote_sasl.c:241
#1 0x00007f7cef4884cb in pni_sasl_impl_free (transport=0x7f7cd4015180)
at /home/mick/latest/qpid-proton-0.26.0/c/src/sasl/sasl.c:181
#2 pn_sasl_free (transport=0x7f7cd4015180)
at /home/mick/latest/qpid-proton-0.26.0/c/src/sasl/sasl.c:764
#3 0x00007f7cef480b90 in pn_transport_finalize (object=0x7f7cd4015180)
at /home/mick/latest/qpid-proton-0.26.0/c/src/core/transport.c:665
#4 0x00007f7cef472a99 in pn_class_decref (clazz=0x7f7cef69aca0 <clazz>,
clazz@entry=0x7f7cef69a520 <PN_OBJECT>, object=0x7f7cd4015180)
at /home/mick/latest/qpid-proton-0.26.0/c/src/core/object/object.c:95
#5 0x00007f7cef472cbf in pn_decref (object=<optimized out>)
at /home/mick/latest/qpid-proton-0.26.0/c/src/core/object/object.c:253
#6 0x00007f7cef480851 in pn_transport_free (transport=<optimized out>)
at /home/mick/latest/qpid-proton-0.26.0/c/src/core/transport.c:644
#7 0x00007f7cef47b994 in pn_connection_driver_destroy (d=d@entry=0x7f7cd4014d98)
at /home/mick/latest/qpid-proton-0.26.0/c/src/core/connection_driver.c:94
#8 0x00007f7cef25b604 in pconnection_final_free (pc=0x7f7cd40147f0)
at /home/mick/latest/qpid-proton-0.26.0/c/src/proactor/epoll.c:889
#9 0x00007f7cef25c4fc in pconnection_cleanup (pc=<optimized out>)
at /home/mick/latest/qpid-proton-0.26.0/c/src/proactor/epoll.c:905
#10 0x00007f7cef25d295 in pconnection_process (pc=0x7f7cd40147f0, events=<optimized out>,
timeout=timeout@entry=false, topup=false, is_io_2=<optimized out>)
at /home/mick/latest/qpid-proton-0.26.0/c/src/proactor/epoll.c:1273
#11 0x00007f7cef25dd03 in proactor_do_epoll (p=0x1ee9600, can_block=can_block@entry=true)
at /home/mick/latest/qpid-proton-0.26.0/c/src/proactor/epoll.c:2139
#12 0x00007f7cef25ef2a in pn_proactor_wait (p=<optimized out>)
at /home/mick/latest/qpid-proton-0.26.0/c/src/proactor/epoll.c:2157
#13 0x00007f7cef7057af in thread_run (arg=0x1db7960)
at /home/mick/latest/qpid-dispatch-1.6.0/src/server.c:994
#14 0x00007f7cef04150b in start_thread () from /lib64/libpthread.so.0
#15 0x00007f7cee30538f in clone () from /lib64/libc.so.6
##############################################################
# Type 3
##############################################################
#0 qd_hash_internal_retrieve_with_hash (hash=<optimized out>,
key=key@entry=0x7f140c097ad8, h=<optimized out>, h=<optimized out>)
at /home/mick/latest/qpid-dispatch-1.6.0/src/hash.c:204
#1 0x00007f1432401a15 in qd_hash_internal_retrieve (key=0x7f140c097ad8, h=0x7f141c000bc0)
at /home/mick/latest/qpid-dispatch-1.6.0/src/hash.c:219
#2 qd_hash_retrieve (h=0x7f141c000bc0, key=key@entry=0x7f140c097ad8,
val=val@entry=0x7ffe6c6ac638) at /home/mick/latest/qpid-dispatch-1.6.0/src/hash.c:270
#3 0x00007f14324312e6 in qdr_lookup_terminus_address_CT (core=0xb656c0,
dir=<optimized out>, conn=conn@entry=0x7f140c076798, terminus=0x7f140c086258,
link_route=link_route@entry=0x7ffe6c6ac77d,
unavailable=unavailable@entry=0x7ffe6c6ac77e, core_endpoint=0x7ffe6c6ac77f,
accept_dynamic=true, create_if_not_found=true)
at /home/mick/latest/qpid-dispatch-1.6.0/src/router_core/modules/address_lookup_client/lookup_client.c:266
#4 0x00007f14324314a8 in qcm_addr_lookup_local_search (
request=request@entry=0x7f141c071e18, client=<optimized out>)
at /home/mick/latest/qpid-dispatch-1.6.0/src/router_core/modules/address_lookup_client/lookup_client.c:415
#5 0x00007f1432431b06 in on_state (core=<optimized out>, api_client=<optimized out>,
user_context=0x7f141c021a90, active=<optimized out>)
at /home/mick/latest/qpid-dispatch-1.6.0/src/router_core/modules/address_lookup_client/lookup_client.c:568
#6 0x00007f143241bc82 in _state_updated_CT (client=0x7f141c065958)
at /home/mick/latest/qpid-dispatch-1.6.0/src/router_core/core_client_api.c:382
#7 0x00007f143241c544 in _sender_detached_CT (client_context=0x7f141c065958, error=0x0)
at /home/mick/latest/qpid-dispatch-1.6.0/src/router_core/core_client_api.c:612
#8 0x00007f143241bb46 in qdrc_endpoint_do_cleanup_CT (core=core@entry=0xb656c0,
ep=0x7f141c062898)
at /home/mick/latest/qpid-dispatch-1.6.0/src/router_core/core_link_endpoint.c:233
#9 0x00007f1432425652 in qdr_core_free (core=0xb656c0)
at /home/mick/latest/qpid-dispatch-1.6.0/src/router_core/router_core.c:168
#10 0x00007f1432434b43 in qd_router_free (router=0xa93b70)
at /home/mick/latest/qpid-dispatch-1.6.0/src/router_node.c:1741
#11 0x00007f143240043c in qd_dispatch_free (qd=0x76b410)
at /home/mick/latest/qpid-dispatch-1.6.0/src/dispatch.c:359
#12 0x0000000000401886 in main_process (
config_path=0x7ffe6c6adff6 "/home/mick/mercury/mercury/sessions/session_2019_03_27_0853/config/edge_0013.conf", python_pkgdir=<optimized out>, test_hooks=<optimized out>, fd=2)
at /home/mick/latest/qpid-dispatch-1.6.0/router/src/main.c:116
#13 0x0000000000401637 in main (argc=5, argv=0x7ffe6c6ac998)
at /home/mick/latest/qpid-dispatch-1.6.0/router/src/main.c:369
##############################################################
# Type 4
##############################################################
#0 pn_collector_put (collector=0x4242424242424242, clazz=0x7fc408147520 <PN_OBJECT>,
context=0x0, type=type@entry=PN_CONNECTION_WAKE)
at /home/mick/latest/qpid-proton-0.26.0/c/src/core/event.c:134
#1 0x00007fc4081b5258 in http_thread_run (v=0x2651530)
at /home/mick/latest/qpid-dispatch-1.6.0/src/http-libwebsockets.c:731
#2 0x00007fc407aee50b in start_thread () from /lib64/libpthread.so.0
#3 0x00007fc406db238f in clone () from /lib64/libc.so.6
##############################################################
# Type 5
##############################################################
#0 0x00007f64f211e0a3 in _int_malloc () from /lib64/libc.so.6
#1 0x00007f64f212010d in malloc () from /lib64/libc.so.6
#2 0x00007f64f33032b0 in pn_buffer (capacity=capacity@entry=64)
at /home/mick/latest/qpid-proton-0.26.0/c/src/core/buffer.c:48
#3 0x00007f64f3304879 in pn_data (capacity=capacity@entry=0)
at /home/mick/latest/qpid-proton-0.26.0/c/src/core/codec.c:374
#4 0x00007f64f35781e4 in qdr_error_from_pn (pn=0x7f64cc0bb0c0)
at /home/mick/latest/qpid-dispatch-1.6.0/src/router_core/error.c:48
#5 0x00007f64f358e51b in AMQP_disposition_handler (context=0xf25880, link=0x7f64c808bdd8,
pnd=0x7f64cc0bb080) at /home/mick/latest/qpid-dispatch-1.6.0/src/router_node.c:676
#6 0x00007f64f355a372 in do_updated (pnd=0x7f64cc0bb080)
at /home/mick/latest/qpid-dispatch-1.6.0/src/container.c:214
#7 qd_container_handle_event (container=<optimized out>, event=event@entry=0x142fb90,
conn=conn@entry=0x7f64c8189990, qd_conn=qd_conn@entry=0x7f64c8002c18)
at /home/mick/latest/qpid-dispatch-1.6.0/src/container.c:643
#8 0x00007f64f3592caa in handle (qd_server=qd_server@entry=0xf74890, e=e@entry=0x142fb90,
pn_conn=0x7f64c8189990, ctx=ctx@entry=0x7f64c8002c18)
at /home/mick/latest/qpid-dispatch-1.6.0/src/server.c:984
#9 0x00007f64f35937e9 in thread_run (arg=arg@entry=0xf74890)
at /home/mick/latest/qpid-dispatch-1.6.0/src/server.c:1009
#10 0x00007f64f3593af0 in qd_server_run (qd=<optimized out>)
at /home/mick/latest/qpid-dispatch-1.6.0/src/server.c:1283
#11 0x000000000040186f in main_process (
config_path=0x7ffd7088d018 "/home/mick/mercury/mercury/sessions/session_2019_04_02_1321/config/D.conf", python_pkgdir=<optimized out>, test_hooks=<optimized out>, fd=2)
at /home/mick/latest/qpid-dispatch-1.6.0/router/src/main.c:112
#12 0x0000000000401637 in main (argc=5, argv=0x7ffd7088c4f8)
at /home/mick/latest/qpid-dispatch-1.6.0/router/src/main.c:369
> Various crashes in 1.6 release
> ------------------------------
>
> Key: DISPATCH-1309
> URL: https://issues.apache.org/jira/browse/DISPATCH-1309
> Project: Qpid Dispatch
> Issue Type: Bug
> Affects Versions: 1.6.0
> Environment: System 'unused':(
> Fedora 5.0.3-200.fc29.x86_64,
> Python 2.7.15,
> Proton master @ eab1f.
> System 'taj':(
> Fedora 4.18.16-200.fc28.x86_64,
> Python 3.6.6,
> Proton master @ 68b38
> Reporter: Chuck Rolke
> Priority: Major
> Attachments: DISPATCH-1309-backtraces.txt, DISPATCH-1309-gen_configs_linear.py
>
>
> qpid-dispatch master @ 51244, which is very close to the 1.6 release, has various crashes.
> The test network is 12 routers spread over two systems. (Configuration generator to be attached.) Four interior routers are in linear arrangement with A and C on one system ('unused'), and B and D on the other system ('taj'). Each system then attaches four edge routers, one to each interior router.
> Running lightweight tests, like proton cpp simple_send and simple_recv to ports on INTA and INTB interior routers leads to a crash on INTC. The crashes typically look like reuse of structures after they have been freed (addresses are 0x9999999999999999). Other crashes hint of general memory corruption (crashes in malloc.c).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org