You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Aaron Fabbri <aj...@gmail.com> on 2010/10/28 09:18:38 UTC
crash in qpid::broker::ExchangeRegistry::registerType? [was: Re:
segfault in Rdma broker]
I did a little debugging and it seems like the first stack trace I
posted was wrong. I was using the wrong libtool binary I guess. Here
is what gdb reports on the correct binary.
I'll try and svn up and retest again tomorrow, but if anyone knows of
any changes in this codepath recently let me know.
#0 0x000000303a89c419 in std::basic_string<char,
std::char_traits<char>, std::allocator<char>
>::compare(std::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&) const () from /usr/lib64/libstdc++.so.6
#1 0x00007f3f00e59a18 in std::operator< <char,
std::char_traits<char>, std::allocator<char> > (__lhs=Traceback (most
recent call last):
File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line
549, in to_string
return self.val['_M_dataplus']['_M_p'].lazy_string (length = len)
RuntimeError: Cannot access memory at address 0xffffffffffffffe8
, __rhs="replication") at
/usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/bits/basic_string.h:2317
#2 0x00007f3f00e59eb5 in std::less<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > >::operator()
(this=0x6fce88, __x=Traceback (most recent call last):
File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line
549, in to_string
return self.val['_M_dataplus']['_M_p'].lazy_string (length = len)
RuntimeError: Cannot access memory at address 0xffffffffffffffe8
, __y="replication") at
/usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/bits/stl_function.h:230
#3 0x00007f3f00f029ba in std::_Rb_tree<std::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::pair<std::basic_string<char, std::char_traits<char>,
std::allocator<char> > const,
boost::function5<boost::shared_ptr<qpid::broker::Exchange>,
std::string const&, bool, qpid::framing::FieldTable const&,
qpid::management::Manageable*, qpid::broker::Broker*> >,
std::_Select1st<std::pair<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const,
boost::function5<boost::shared_ptr<qpid::broker::Exchange>,
std::string const&, bool, qpid::framing::FieldTable const&,
qpid::management::Manageable*, qpid::broker::Broker*> > >,
std::less<std::basic_string<char, std::char_traits<char>,
std::allocator<char> > >,
std::allocator<std::pair<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const,
boost::function5<boost::shared_ptr<qpid::broker::Exchange>,
std::string const&, bool, qpid::framing::FieldTable const&,
qpid::management::Manageable*, qpid::broker::Broker*> > >
>::_M_insert_unique_ (this=0x6fce88, __position=..., __v=...)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/bits/stl_tree.h:1212
#4 0x00007f3f00f01cac in std::map<std::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
boost::function5<boost::shared_ptr<qpid::broker::Exchange>,
std::string const&, bool, qpid::framing::FieldTable const&,
qpid::management::Manageable*, qpid::broker::Broker*>,
std::less<std::basic_string<char, std::char_traits<char>,
std::allocator<char> > >,
std::allocator<std::pair<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const,
boost::function5<boost::shared_ptr<qpid::broker::Exchange>,
std::string const&, bool, qpid::framing::FieldTable const&,
qpid::management::Manageable*, qpid::broker::Broker*> > > >::insert
(this=0x6fce88,
__position=..., __x=...) at
/usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/bits/stl_map.h:540
#5 0x00007f3f00f011c0 in std::map<std::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
boost::function5<boost::shared_ptr<qpid::broker::Exchange>,
std::string const&, bool, qpid::framing::FieldTable const&,
qpid::management::Manageable*, qpid::broker::Broker*>,
std::less<std::basic_string<char, std::char_traits<char>,
std::allocator<char> > >,
std::allocator<std::pair<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const,
boost::function5<boost::shared_ptr<qpid::broker::Exchange>,
std::string const&, bool, qpid::framing::FieldTable const&,
qpid::management::Manageable*, qpid::broker::Broker*> > >
>::operator[] (this=0x6fce88,
__k="replication") at
/usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/bits/stl_map.h:450
#6 0x00007f3f00f006c5 in qpid::broker::ExchangeRegistry::registerType
(this=0x6fce58, type="replication", f=...) at
../../src/qpid/broker/ExchangeRegistry.cpp:103
#7 0x00007f3efedbaff9 in
qpid::replication::ReplicationExchangePlugin::earlyInitialize
(this=0x7f3efefc1ce0, target=<value optimized out>) at
qpid/replication/ReplicationExchange.cpp:223
#8 0x00007f3f008e8a5e in boost::_mfi::mf1<void, qpid::Plugin,
qpid::Plugin::Target&>::operator() (this=0x7fff46858c30,
p=0x7f3efefc1ce0, a1=...) at
/usr/include/boost/bind/mem_fn_template.hpp:162
#9 0x00007f3f008e8506 in boost::_bi::list2<boost::arg<1>,
boost::reference_wrapper<qpid::Plugin::Target>
>::operator()<boost::_mfi::mf1<void, qpid::Plugin,
qpid::Plugin::Target&>, boost::_bi::list1<qpid::Plugin* const&> > (
this=0x7fff46858c40, f=..., a=...) at /usr/include/boost/bind/bind.hpp:306
#10 0x00007f3f008e7cd2 in boost::_bi::bind_t<void,
boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>,
boost::_bi::list2<boost::arg<1>,
boost::reference_wrapper<qpid::Plugin::Target> >
>::operator()<qpid::Plugin*> (
this=0x7fff46858c30, a1=@0x6ffb68) at
/usr/include/boost/bind/bind_template.hpp:47
#11 0x00007f3f008e721c in
std::for_each<__gnu_cxx::__normal_iterator<qpid::Plugin* const*,
std::vector<qpid::Plugin*, std::allocator<qpid::Plugin*> > >,
boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::Plugin,
qpid::Plugin::Target&>, boost::_bi::list2<boost::arg<1>,
boost::reference_wrapper<qpid::Plugin::Target> > > > (__first=,
__last=, __f=...) at
/usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/bits/stl_algo.h:4200
#12 0x00007f3f008e5da4 in qpid::(anonymous
namespace)::each_plugin<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, qpid::Plugin, qpid::Plugin::Target&>,
boost::_bi::list2<boost::arg<1>,
boost::reference_wrapper<qpid::Plugin::Target> > > > (f=...) at
../../src/qpid/Plugin.cpp:73
#13 0x00007f3f008e5cdf in qpid::Plugin::earlyInitAll (t=...) at
../../src/qpid/Plugin.cpp:87
#14 0x00007f3f00eb1830 in qpid::broker::Broker::Broker (this=0x6fcb90,
conf=...) at ../../src/qpid/broker/Broker.cpp:223
#15 0x0000000000413b4c in QpiddBroker::execute (this=0x7fff46859b9f,
options=0x6f56b0) at ../../src/posix/QpiddBroker.cpp:182
#16 0x0000000000411593 in main (argc=13, argv=0x7fff46859e68) at
../../src/qpidd.cpp:80
On Wed, Oct 27, 2010 at 6:15 PM, Steve Huston <sh...@riverace.com> wrote:
> Hi Aaron,
>
>> I recently updated to svn trunk latest and now my Rdma broker
>> is crashing.
>>
>> 1. Is anyone else seeing this?
>
> I don't recall any other reports on this list.
>
>> 2. Should I open a bug.. and if so please give pointer to url
>> and anything else I'd need to know.
>
> Yes, please. The URL is http://issues.apache.org/jira/browse/qpid
>
> Hopefully Andrew will weigh in on your stack... He's the rdma wiz.
>
> Thanks,
> -Steve
>
>> I'm running from gdb with this script
>>
>> # cat gdbscript
>> file /aafabbri/autotools_build/src/.libs/qpidd
>>
>> # get libraries loaded
>> break main
>> run
>> del 1
>>
>> define go
>> run \
>> --auth no \
>> --mgmt-enable no \
>> --load-module /aafabbri/autotools_build/src/.libs/rdma.so \
>> --transport rdma \
>> --worker-threads 4 \
>> --log-to-stdout yes
>> end
>>
>> # LD_LIBRARY_PATH=/aafabbri/autotools_build/src/.libs gdb -x
>> myscript ...
>> (gdb) go
>> [Thread debugging using libthread_db enabled]
>> [New Thread 0x7ffff5af0710 (LWP 2850)]
>> 2010-10-27 15:28:32 notice SASL disabled: No Authentication
>> Performed 2010-10-27 15:28:32 notice SASL disabled: No
>> Authentication Performed 2010-10-27 15:28:32 notice Listening
>> on TCP port 5672 2010-10-27 15:28:32 notice Listening on TCP
>> port 5672 2010-10-27 15:28:32 notice Rdma: Listening on RDMA
>> port 5672 2010-10-27 15:28:32 notice Rdma: Listening on RDMA
>> port 5672 2010-10-27 15:28:32 notice Rdma: Listening on RDMA
>> port 5672 2010-10-27 15:28:32 notice Rdma: Listening on RDMA
>> port 5672 5672 2010-10-27 15:28:32 notice Broker running
>> 2010-10-27 15:28:32 notice Broker running [New Thread
>> 0x7ffff4cd6710 (LWP 2851)] [New Thread 0x7ffff42d5710 (LWP
>> 2852)] [New Thread 0x7ffff38d4710 (LWP 2853)]
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 0x7ffff38d4710 (LWP 2853)]
>> 0x00007ffff67cacb2 in operator() (function_obj_ptr=<value optimized
>> out>, a0=...)
>> at /usr/include/boost/bind/mem_fn_template.hpp:49
>> 49 BOOST_MEM_FN_RETURN (p->*f_)();
>> (gdb) thread apply all bt
>>
>> Thread 5 (Thread 0x7ffff38d4710 (LWP 2853)):
>> #0 0x00007ffff67cacb2 in operator() (function_obj_ptr=<value
>> optimized out>, a0=...)
>> at /usr/include/boost/bind/mem_fn_template.hpp:49
>> #1 operator()<boost::_mfi::mf0<void, Rdma::AsynchIO>,
>> boost::_bi::list1<qpid::sys::DispatchHandle&> > (
>> function_obj_ptr=<value optimized out>, a0=...) at
>> /usr/include/boost/bind/bind.hpp:246
>> #2 operator()<qpid::sys::DispatchHandle>
>> (function_obj_ptr=<value optimized out>, a0=...)
>> at /usr/include/boost/bind/bind_template.hpp:32
>> #3
>> boost::detail::function::void_function_obj_invoker1<boost::_bi
>> ::bind_t<void,
>> boost::_mfi::mf0<void, Rdma::AsynchIO>,
>> boost::_bi::list1<boost::_bi::value<Rdma::AsynchIO*> > >,
>> void, qpid::sys::DispatchHandle&>::invoke (
>> function_obj_ptr=<value optimized out>, a0=...) at
>> /usr/include/boost/function/function_template.hpp:153
>> #4 0x00007ffff78f5298 in boost::function1<void,
>> qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>,
>> a0=<value optimized out>) at
>> /usr/include/boost/function/function_template.hpp:1013
>> #5 0x00007ffff78f45b9 in qpid::sys::DispatchHandle::processEvent
>> (this=0x7fffe80021a0,
>> type=qpid::sys::Poller::READABLE) at
>> qpid/sys/DispatchHandle.cpp:278 #6 0x00007ffff78450d2 in
>> process (this=0x62fc50) at ./qpid/sys/Poller.h:131 #7
>> qpid::sys::Poller::run (this=0x62fc50) at
>> qpid/sys/epoll/EpollPoller.cpp:519
>> #8 0x00007ffff783cf6a in qpid::sys::(anonymous
>> namespace)::runRunnable (p=<value optimized out>)
>> at qpid/sys/posix/Thread.cpp:35
>> #9 0x000000305a207761 in start_thread () from
>> /lib64/libpthread.so.0 #10 0x0000003059ee14fd in clone ()
>> from /lib64/libc.so.6
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: crash in qpid::broker::ExchangeRegistry::registerType? [was: Re:
segfault in Rdma broker]
Posted by Aaron Fabbri <aj...@gmail.com>.
Yep you got it. Thanks for the hint. Adding --no-module-dir (also
suggested by Andrew Stitcher) fixed it. I'm going to file a bug and
you guys can decide what (if anything) to do about it.
On Thu, Oct 28, 2010 at 2:27 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 10/28/2010 08:18 AM, Aaron Fabbri wrote:
>>
>> I did a little debugging and it seems like the first stack trace I
>> posted was wrong. I was using the wrong libtool binary I guess. Here
>> is what gdb reports on the correct binary.
>>
>> I'll try and svn up and retest again tomorrow, but if anyone knows of
>> any changes in this codepath recently let me know.
>
> Do you have any 'old' modules installed perhaps (generally under
> /usr/lib[64]/qpid/daemon)? The ABI for plugins is not stable at all so if
> you have older binaries and start a new broker that can cause problems.
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: crash in qpid::broker::ExchangeRegistry::registerType? [was:
Re: segfault in Rdma broker]
Posted by Gordon Sim <gs...@redhat.com>.
On 10/28/2010 08:18 AM, Aaron Fabbri wrote:
> I did a little debugging and it seems like the first stack trace I
> posted was wrong. I was using the wrong libtool binary I guess. Here
> is what gdb reports on the correct binary.
>
> I'll try and svn up and retest again tomorrow, but if anyone knows of
> any changes in this codepath recently let me know.
Do you have any 'old' modules installed perhaps (generally under
/usr/lib[64]/qpid/daemon)? The ABI for plugins is not stable at all so
if you have older binaries and start a new broker that can cause problems.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org