You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Gordon Sim <gs...@redhat.com> on 2015/07/01 21:27:40 UTC
proton segfault on freeing collector? (was Re: qpidd 0.32 crashes)
On 06/30/2015 10:17 AM, Michael Ivanov wrote:
> Ok it crashed again and here are the backtrace and protocol trace:
The error occurs when draining events on the collector as part of
freeing it. The connection has already been 'freed' by the application,
but is referenced by an event in the collector. My understanding is the
reference counting scheme used by proton is supposed to handle this.
Unfortunately I'm still unable to reproduce this. I have tried both
simple tests that generate events then free the connection then the
associated collector with events still in it and that all seems to work
fine. I've also tried some stress tests where the transport is killed
while the client is doing various things (that seems to be what the
protocol trace indicates, since there is no explicit close).
The broker could free the collector *before* freeing the connection if
that was required, but from what I can tell it should make no difference.
Can anyone more familiar with proton internals spot anything odd with
regard to the connection state below (after stack trace)? The apparent
location of the segfault is when calling pn_refcount on the connection.
The only reason I can see for that is if the 'head' containing class and
reference count was no longer valid.
Is it possible to reproduce when running the broker under valgrind?
(Sorry to keep asking for more things, but without a reproducer and
extensive knowledge of proton, its hard to figure out what the issue
might be).
> Program received signal SIGSEGV, Segmentation fault.
> 0x000000000115bc00 in ?? ()
> (gdb) bt
> #0 0x000000000115bc00 in ?? ()
> #1 0x00007fb7c6afc74d in pn_connection_finalize (object=0xda2c90) at
> /usr/src/debug/qpid-proton-0.9/proton-c/src/engine/engine.c:467
> #2 0x00007fb7c6af15c8 in pn_class_decref (clazz=0x7fb7c6d25400 <clazz.4691>, object=0xda2c90)
> at /usr/src/debug/qpid-proton-0.9/proton-c/src/object/object.c:97
> #3 0x00007fb7c6af15c8 in pn_class_decref (clazz=0x7fb7c6d25380 <clazz.4874>, object=0xebfc90)
> at /usr/src/debug/qpid-proton-0.9/proton-c/src/object/object.c:97
> #4 0x00007fb7c6af15c8 in pn_class_decref (clazz=0x7fb7c6d25300 <clazz.4927>, object=0x1156450)
> at /usr/src/debug/qpid-proton-0.9/proton-c/src/object/object.c:97
> #5 0x00007fb7c6aff310 in pn_event_finalize (event=0xe33710) at /usr/src/debug/qpid-proton-0.9/proton-c/src/events/event.c:190
> #6 pn_event_finalize_cast (object=0xe33710) at /usr/src/debug/qpid-proton-0.9/proton-c/src/events/event.c:235
> #7 0x00007fb7c6af15c8 in pn_class_decref (clazz=0x7fb7c6d25480 <clazz.2269>, object=0xe33710)
> at /usr/src/debug/qpid-proton-0.9/proton-c/src/object/object.c:97
> #8 0x00007fb7c6af17f2 in pn_decref (object=<optimized out>) at /usr/src/debug/qpid-proton-0.9/proton-c/src/object/object.c:252
> #9 0x00007fb7c6aff4b2 in pn_collector_pop (collector=collector@entry=0xdac360) at
> /usr/src/debug/qpid-proton-0.9/proton-c/src/events/event.c:167
> #10 0x00007fb7c6aff508 in pn_collector_drain (collector=0xdac360) at /usr/src/debug/qpid-proton-0.9/proton-c/src/events/event.c:34
> #11 pn_collector_release (collector=collector@entry=0xdac360) at /usr/src/debug/qpid-proton-0.9/proton-c/src/events/event.c:96
> #12 0x00007fb7c6aff529 in pn_collector_free (collector=0xdac360) at /usr/src/debug/qpid-proton-0.9/proton-c/src/events/event.c:87
> #13 0x00007fb7c6d79669 in qpid::broker::amqp::Connection::~Connection (this=0xda2ec0, __in_chrg=<optimized out>)
> at /usr/src/debug/qpid-cpp-0.32/src/qpid/broker/amqp/Connection.cpp:171
> #14 0x00007fb7c6d79819 in qpid::broker::amqp::Connection::~Connection (this=0xda2ec0, __in_chrg=<optimized out>)
> at /usr/src/debug/qpid-cpp-0.32/src/qpid/broker/amqp/Connection.cpp:173
> #15 0x00007fb7ca57fd63 in qpid::sys::AsynchIOHandler::~AsynchIOHandler (this=0xda2920, __in_chrg=<optimized out>)
> at /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/AsynchIOHandler.cpp:73
> #16 0x00007fb7ca57fe59 in qpid::sys::AsynchIOHandler::~AsynchIOHandler (this=0xda2920, __in_chrg=<optimized out>)
> at /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/AsynchIOHandler.cpp:74
> #17 0x00007fb7ca580e5f in qpid::sys::AsynchIOHandler::closedSocket (this=0xda2920, s=...)
> at /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/AsynchIOHandler.cpp:196
> #18 0x00007fb7ca4fa43c in qpid::sys::posix::AsynchIO::writeable (this=0xebb420, h=...)
> at /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/posix/AsynchIO.cpp:575
> #19 0x00007fb7ca5847a1 in operator() (a0=..., this=<optimized out>) at /usr/include/boost/function/function_template.hpp:767
> #20 qpid::sys::DispatchHandle::processEvent (this=0xebb428, type=qpid::sys::Poller::WRITABLE)
> at /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/DispatchHandle.cpp:283
> #21 0x00007fb7ca52258e in process (this=<synthetic pointer>, this=<synthetic pointer>) at
> /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/Poller.h:131
> #22 qpid::sys::Poller::run (this=0xd6ff30) at /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/epoll/EpollPoller.cpp:522
> #23 0x00007fb7caab8238 in qpid::broker::Broker::run (this=0xd72d90) at /usr/src/debug/qpid-cpp-0.32/src/qpid/broker/Broker.cpp:522
> #24 0x00000000004055cb in qpid::broker::QpiddBroker::execute (this=this@entry=0x7fffaba3ad3e, options=<optimized out>)
> at /usr/src/debug/qpid-cpp-0.32/src/posix/QpiddBroker.cpp:214
> #25 0x0000000000409004 in qpid::broker::run_broker (argc=5, argv=0x7fffaba3b0d8, hidden=<optimized out>)
> at /usr/src/debug/qpid-cpp-0.32/src/qpidd.cpp:108
> #26 0x00007fb7c95d4af5 in __libc_start_main (main=0x404750 <main(int, char**)>, argc=5, ubp_av=0x7fffaba3b0d8, init=<optimized
> out>,
> fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffaba3b0c8) at libc-start.c:274
> #27 0x00000000004049c1 in _start ()
> (gdb)
> (gdb) fr 1
> #1 0x00007fb7c6afc74d in pn_connection_finalize (object=0xda2c90) at
> /usr/src/debug/qpid-proton-0.9/proton-c/src/engine/engine.c:467
> 467 if (pn_refcount(conn) > 0) {
> (gdb) p *conn
> $1 = {endpoint = {type = CONNECTION, state = 18, error = 0xda3bb0, condition = {name = 0xda2db0, description = 0xda2de0, info =
> 0xda2e10},
> remote_condition = {name = 0xebb870, description = 0xda3180, info = 0xda31d0}, endpoint_next = 0xdac6c0, endpoint_prev = 0x0,
> transport_next = 0x0, transport_prev = 0x0, refcount = 0, modified = false, freed = true, referenced = true}, endpoint_head
> = 0x0,
> endpoint_tail = 0x0, transport_head = 0x0, transport_tail = 0x0, sessions = 0xda2b00, freed = 0xda4250, transport = 0x0,
> work_head = 0x0,
> work_tail = 0x0, tpwork_head = 0x0, tpwork_tail = 0x0, container = 0xe199b0, hostname = 0xe19a00, offered_capabilities =
> 0xda5b50,
> desired_capabilities = 0xda5df0, properties = 0xda5fb0, collector = 0xdac360, context = 0xda61c0, delivery_pool = 0xda6210}
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: proton segfault on freeing collector? (was Re: qpidd 0.32 crashes)
Posted by Ted Ross <tr...@redhat.com>.
On 07/01/2015 03:27 PM, Gordon Sim wrote:
> On 06/30/2015 10:17 AM, Michael Ivanov wrote:
>> Ok it crashed again and here are the backtrace and protocol trace:
>
> The error occurs when draining events on the collector as part of
> freeing it. The connection has already been 'freed' by the application,
> but is referenced by an event in the collector. My understanding is the
> reference counting scheme used by proton is supposed to handle this.
>
> Unfortunately I'm still unable to reproduce this. I have tried both
> simple tests that generate events then free the connection then the
> associated collector with events still in it and that all seems to work
> fine. I've also tried some stress tests where the transport is killed
> while the client is doing various things (that seems to be what the
> protocol trace indicates, since there is no explicit close).
>
> The broker could free the collector *before* freeing the connection if
> that was required, but from what I can tell it should make no difference.
For what it's worth, there was a crash in Dispatch which seemed to be
caused by freeing the collector before the connection. See
https://svn.apache.org/viewvc?view=revision&revision=1588363
>
> Can anyone more familiar with proton internals spot anything odd with
> regard to the connection state below (after stack trace)? The apparent
> location of the segfault is when calling pn_refcount on the connection.
> The only reason I can see for that is if the 'head' containing class and
> reference count was no longer valid.
>
> Is it possible to reproduce when running the broker under valgrind?
> (Sorry to keep asking for more things, but without a reproducer and
> extensive knowledge of proton, its hard to figure out what the issue
> might be).
>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x000000000115bc00 in ?? ()
>> (gdb) bt
>> #0 0x000000000115bc00 in ?? ()
>> #1 0x00007fb7c6afc74d in pn_connection_finalize (object=0xda2c90) at
>> /usr/src/debug/qpid-proton-0.9/proton-c/src/engine/engine.c:467
>> #2 0x00007fb7c6af15c8 in pn_class_decref (clazz=0x7fb7c6d25400
>> <clazz.4691>, object=0xda2c90)
>> at /usr/src/debug/qpid-proton-0.9/proton-c/src/object/object.c:97
>> #3 0x00007fb7c6af15c8 in pn_class_decref (clazz=0x7fb7c6d25380
>> <clazz.4874>, object=0xebfc90)
>> at /usr/src/debug/qpid-proton-0.9/proton-c/src/object/object.c:97
>> #4 0x00007fb7c6af15c8 in pn_class_decref (clazz=0x7fb7c6d25300
>> <clazz.4927>, object=0x1156450)
>> at /usr/src/debug/qpid-proton-0.9/proton-c/src/object/object.c:97
>> #5 0x00007fb7c6aff310 in pn_event_finalize (event=0xe33710) at
>> /usr/src/debug/qpid-proton-0.9/proton-c/src/events/event.c:190
>> #6 pn_event_finalize_cast (object=0xe33710) at
>> /usr/src/debug/qpid-proton-0.9/proton-c/src/events/event.c:235
>> #7 0x00007fb7c6af15c8 in pn_class_decref (clazz=0x7fb7c6d25480
>> <clazz.2269>, object=0xe33710)
>> at /usr/src/debug/qpid-proton-0.9/proton-c/src/object/object.c:97
>> #8 0x00007fb7c6af17f2 in pn_decref (object=<optimized out>) at
>> /usr/src/debug/qpid-proton-0.9/proton-c/src/object/object.c:252
>> #9 0x00007fb7c6aff4b2 in pn_collector_pop
>> (collector=collector@entry=0xdac360) at
>> /usr/src/debug/qpid-proton-0.9/proton-c/src/events/event.c:167
>> #10 0x00007fb7c6aff508 in pn_collector_drain (collector=0xdac360) at
>> /usr/src/debug/qpid-proton-0.9/proton-c/src/events/event.c:34
>> #11 pn_collector_release (collector=collector@entry=0xdac360) at
>> /usr/src/debug/qpid-proton-0.9/proton-c/src/events/event.c:96
>> #12 0x00007fb7c6aff529 in pn_collector_free (collector=0xdac360) at
>> /usr/src/debug/qpid-proton-0.9/proton-c/src/events/event.c:87
>> #13 0x00007fb7c6d79669 in qpid::broker::amqp::Connection::~Connection
>> (this=0xda2ec0, __in_chrg=<optimized out>)
>> at
>> /usr/src/debug/qpid-cpp-0.32/src/qpid/broker/amqp/Connection.cpp:171
>> #14 0x00007fb7c6d79819 in qpid::broker::amqp::Connection::~Connection
>> (this=0xda2ec0, __in_chrg=<optimized out>)
>> at
>> /usr/src/debug/qpid-cpp-0.32/src/qpid/broker/amqp/Connection.cpp:173
>> #15 0x00007fb7ca57fd63 in qpid::sys::AsynchIOHandler::~AsynchIOHandler
>> (this=0xda2920, __in_chrg=<optimized out>)
>> at /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/AsynchIOHandler.cpp:73
>> #16 0x00007fb7ca57fe59 in qpid::sys::AsynchIOHandler::~AsynchIOHandler
>> (this=0xda2920, __in_chrg=<optimized out>)
>> at /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/AsynchIOHandler.cpp:74
>> #17 0x00007fb7ca580e5f in qpid::sys::AsynchIOHandler::closedSocket
>> (this=0xda2920, s=...)
>> at /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/AsynchIOHandler.cpp:196
>> #18 0x00007fb7ca4fa43c in qpid::sys::posix::AsynchIO::writeable
>> (this=0xebb420, h=...)
>> at /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/posix/AsynchIO.cpp:575
>> #19 0x00007fb7ca5847a1 in operator() (a0=..., this=<optimized out>) at
>> /usr/include/boost/function/function_template.hpp:767
>> #20 qpid::sys::DispatchHandle::processEvent (this=0xebb428,
>> type=qpid::sys::Poller::WRITABLE)
>> at /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/DispatchHandle.cpp:283
>> #21 0x00007fb7ca52258e in process (this=<synthetic pointer>,
>> this=<synthetic pointer>) at
>> /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/Poller.h:131
>> #22 qpid::sys::Poller::run (this=0xd6ff30) at
>> /usr/src/debug/qpid-cpp-0.32/src/qpid/sys/epoll/EpollPoller.cpp:522
>> #23 0x00007fb7caab8238 in qpid::broker::Broker::run (this=0xd72d90) at
>> /usr/src/debug/qpid-cpp-0.32/src/qpid/broker/Broker.cpp:522
>> #24 0x00000000004055cb in qpid::broker::QpiddBroker::execute
>> (this=this@entry=0x7fffaba3ad3e, options=<optimized out>)
>> at /usr/src/debug/qpid-cpp-0.32/src/posix/QpiddBroker.cpp:214
>> #25 0x0000000000409004 in qpid::broker::run_broker (argc=5,
>> argv=0x7fffaba3b0d8, hidden=<optimized out>)
>> at /usr/src/debug/qpid-cpp-0.32/src/qpidd.cpp:108
>> #26 0x00007fb7c95d4af5 in __libc_start_main (main=0x404750 <main(int,
>> char**)>, argc=5, ubp_av=0x7fffaba3b0d8, init=<optimized
>> out>,
>> fini=<optimized out>, rtld_fini=<optimized out>,
>> stack_end=0x7fffaba3b0c8) at libc-start.c:274
>> #27 0x00000000004049c1 in _start ()
>> (gdb)
>> (gdb) fr 1
>> #1 0x00007fb7c6afc74d in pn_connection_finalize (object=0xda2c90) at
>> /usr/src/debug/qpid-proton-0.9/proton-c/src/engine/engine.c:467
>> 467 if (pn_refcount(conn) > 0) {
>> (gdb) p *conn
>> $1 = {endpoint = {type = CONNECTION, state = 18, error = 0xda3bb0,
>> condition = {name = 0xda2db0, description = 0xda2de0, info =
>> 0xda2e10},
>> remote_condition = {name = 0xebb870, description = 0xda3180, info
>> = 0xda31d0}, endpoint_next = 0xdac6c0, endpoint_prev = 0x0,
>> transport_next = 0x0, transport_prev = 0x0, refcount = 0,
>> modified = false, freed = true, referenced = true}, endpoint_head
>> = 0x0,
>> endpoint_tail = 0x0, transport_head = 0x0, transport_tail = 0x0,
>> sessions = 0xda2b00, freed = 0xda4250, transport = 0x0,
>> work_head = 0x0,
>> work_tail = 0x0, tpwork_head = 0x0, tpwork_tail = 0x0, container =
>> 0xe199b0, hostname = 0xe19a00, offered_capabilities =
>> 0xda5b50,
>> desired_capabilities = 0xda5df0, properties = 0xda5fb0, collector =
>> 0xdac360, context = 0xda61c0, delivery_pool = 0xda6210}
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org