You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Scott Nicol <sn...@tessellationsoftware.com> on 2015/03/30 01:00:26 UTC
C++ Broker crash -- related to QPID-5765?
Hi,
I have been working on a project using qpid, and we have been having
problems with the C++ broker crashing. We're running version 0.28. Based on
the circumstances bug 5765 https://issues.apache.org/jira/browse/QPID-5765
looks like a good possibility. Comments?
Our theory:
1. Database server crashes. Exact time of crash isn't clear, but it was
around the time the broker crashed. If it occurred before the broker crash,
then,,,
2. Persistent store is not available, causing the queues to back up.
3. qpid kills the federation link due to overflowing queue.
4. a few seconds later, broker tries to bind() across a federation link.
This triggers the crash.
>From the qpidd log I see this, 2 seconds before the crash:
2015-03-28 14:47:45 [Store] warning mrg::msgstore::JournalImpl::log:
Journal "CustomerPersistServer-1:PilotCustomerPersistService-1": Enqueue
capacity threshold exceeded on queue
"CustomerPersistServer-1:PilotCustomerPersistService-1".
2015-03-28 14:47:45 [Broker] warning
qpid::broker::<unnamed>::ExInfo::store: Exchange TAP cannot deliver to
queue CustomerPersistServer-1:PilotCustomerPersistService-1: Enqueue
capacity threshold exceeded on queue
"CustomerPersistServer-1:PilotCustomerPersistService-1".
(/pilot/dev/qpid_src/src/qpid-0.28/cpp/src/qpid/legacystore/JournalImpl.cpp:594)
2015-03-28 14:47:45 [Protocol] error
qpid::broker::amqp_0_10::Connection::close: Connection qpid.tcp:
192.168.5.13:8089 closed by error: Enqueue capacity threshold exceeded on
queue "CustomerPersistServer-1:PilotCustomerPersistService-1".
(/pilot/dev/qpid_src/src/qpid-0.28/cpp/src/qpid/legacystore/JournalImpl.cpp:594)(501)
A debug log also picked it up (ignore the hours, they're the same, but one
is in local time, the other gmt)
Sat Mar 28 06:47:45 2015 org.apache.qpid.legacystore:enqThresholdExceeded
what=Journal enqueue capacity threshold exceeded
jrnlId=CustomerPersistServer-1:PilotCustomerPersistService-1
Sat Mar 28 06:47:45 2015 org.apache.qpid.broker:brokerLinkDown rhost=
192.168.5.13:8089
And then the broker crashed, apparently attempting to propagate a bind()
across a federation link, perhaps the one that was closed 2 seconds prior:
#0 std::basic_string<char, std::char_traits<char>, std::allocator<char>
>::find (this=0x7f162abfa280, __str=<value optimized out>, __pos=0)
at
/usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.h:1658
#1 0x00007f16ad8ab070 in
qpid::broker::Bridge::propagateBinding(std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, std::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
qpid::framing::FieldTable*) () from
/pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#2 0x00007f16ad858eb9 in
qpid::broker::Exchange::propagateFedOp(std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, std::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
qpid::framing::FieldTable*) () from
/pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#3 0x00007f16ad9720db in
qpid::broker::TopicExchange::bind(boost::shared_ptr<qpid::broker::Queue>,
std::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, qpid::framing::FieldTable const*) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#4 0x00007f16ad884e50 in
qpid::broker::Queue::bind(boost::shared_ptr<qpid::broker::Exchange>,
std::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, qpid::framing::FieldTable const&) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#5 0x00007f16ad839a3d in
qpid::broker::Broker::bind(std::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, qpid::framing::FieldTable const&, qpid::broker::OwnershipToken
const*, std::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#6 0x00007f16ad951dd1 in
qpid::broker::SessionAdapter::ExchangeHandlerImpl::bind(std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, std::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&, qpid::framing::FieldTable const&) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#7 0x00007f16ad11de2b in void
qpid::framing::ExchangeBindBody::invoke<qpid::framing::AMQP_ServerOperations::ExchangeHandler>(qpid::framing::AMQP_ServerOperations::ExchangeHandler&)
const ()
from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#8 0x00007f16ad11bde1 in
qpid::framing::AMQP_ServerOperations::ExchangeHandler::Invoker::visit(qpid::framing::ExchangeBindBody
const&) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#9 0x00007f16ad102b50 in
qpid::framing::ExchangeBindBody::accept(qpid::framing::MethodBodyConstVisitor&)
const ()
from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#10 0x00007f16ad11a18e in
qpid::framing::AMQP_ServerOperations::Invoker::visit(qpid::framing::ExchangeBindBody
const&) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#11 0x00007f16ad102b50 in
qpid::framing::ExchangeBindBody::accept(qpid::framing::MethodBodyConstVisitor&)
const ()
from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#12 0x00007f16ad963d03 in qpid::framing::Invoker::Result
qpid::framing::invoke<qpid::broker::SessionAdapter>(qpid::broker::SessionAdapter&,
qpid::framing::AMQMethodBody const&) () from
/pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#13 0x00007f16ad95fd8b in
qpid::broker::SessionState::handleCommand(qpid::framing::AMQMethodBody*) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#14 0x00007f16ad960a78 in
qpid::broker::SessionState::handleIn(qpid::framing::AMQFrame&) () from
/pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#15 0x00007f16ad968863 in
qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface,
&(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle(qpid::framing::AMQFrame&)
()
from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#16 0x00007f16ad17faeb in
qpid::amqp_0_10::SessionHandler::handleIn(qpid::framing::AMQFrame&) () from
/pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#17 0x00007f16ad968863 in
qpid::framing::Handler<qpid::framing::AMQFrame&>::MemF---Type <return> to
continue, or q <return> to quit---
unRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface,
&(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle(qpid::framing::AMQFrame&)
()
from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#18 0x00007f16ad8bdcca in
qpid::framing::Handler<qpid::framing::AMQFrame&>::operator()(qpid::framing::AMQFrame&)
()
from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#19 0x00007f16ad8b98d8 in
qpid::broker::ConnectionHandler::handle(qpid::framing::AMQFrame&) () from
/pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#20 0x00007f16ad8b02a8 in
qpid::broker::amqp_0_10::Connection::received(qpid::framing::AMQFrame&) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#21 0x00007f16ad81dd84 in qpid::amqp_0_10::Connection::decode(char const*,
unsigned long) () from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#22 0x00007f16ad92a00d in qpid::broker::SecureConnection::decode(char
const*, unsigned long) () from
/pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#23 0x00007f16ad1b4739 in
qpid::sys::AsynchIOHandler::readbuff(qpid::sys::AsynchIO&,
qpid::sys::AsynchIOBufferBase*) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#24 0x00007f16ad9c71c6 in boost::_mfi::mf2<void,
qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&,
qpid::sys::AsynchIOBufferBase*>::operator()(qpid::sys::AsynchIOHandler*,
qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*) const ()
from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#25 0x00007f16ad9c5f04 in void
boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>,
boost::arg<1>, boost::arg<2> >::operator()<boost::_mfi::mf2<void,
qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&,
qpid::sys::AsynchIOBufferBase*>, boost::_bi::list2<qpid::sys::AsynchIO&,
qpid::sys::AsynchIOBufferBase*&> >(boost::_bi::type<void>,
boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&,
qpid::sys::AsynchIOBufferBase*>&, boost::_bi::list2<qpid::sys::AsynchIO&,
qpid::sys::AsynchIOBufferBase*&>&, int) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#26 0x00007f16ad9c5396 in void boost::_bi::bind_t<void,
boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&,
qpid::sys::AsynchIOBufferBase*>,
boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>,
boost::arg<1>, boost::arg<2> > >::operator()<qpid::sys::AsynchIO,
qpid::sys::AsynchIOBufferBase*>(qpid::sys::AsynchIO&,
qpid::sys::AsynchIOBufferBase*&) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#27 0x00007f16ad9c462f in
boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void,
boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&,
qpid::sys::AsynchIOBufferBase*>,
boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>,
boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&,
qpid::sys::AsynchIOBufferBase*>::invoke(boost::detail::function::function_buffer&,
qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidbroker.so.2
#28 0x00007f16ad12ccb0 in boost::function2<void, qpid::sys::AsynchIO&,
qpid::sys::AsynchIOBufferBase*>::operator()(qpid::sys::AsynchIO&,
qpid::sys::AsynchIOBufferBase*) const () from
/pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#29 0x00007f16ad129da0 in
qpid::sys::posix::AsynchIO::readable(qpid::sys::DispatchHandle&) () from
/pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#30 0x00007f16ad131cae in boost::_mfi::mf1<void,
qpid::sys::posix::AsynchIO,
qpid::sys::DispatchHandle&>::operator()(qpid::sys::posix::AsynchIO*,
qpid::sys::DispatchHandle&) const () from
/pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#31 0x00007f16ad130fff in void
boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>,
boost::arg<1> >::operator()<boost::_mfi::mf1<void,
qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>,
boost::_bi::list1<qpid::sys::DispatchHandle&> >(boost::_bi::type<void>,
boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO,
qpid::sys::DispatchHandle&>&,
boost::_bi::list1<qpid::sys::DispatchHandle&>&, int) () from
/pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#32 0x00007f16ad1302f0 in void boost::_bi::bind_t<void,
boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO,
qpid::sys::DispatchHandle&>,
boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>,
boost::arg<1> >
>::operator()<qpid::sys::DispatchHandle>(qpid::sys::DispatchHandle&) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#33 0x00007f16ad12f2cd in
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO,
qpid::sys::DispatchHandle&>,
boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>,
boost::arg<1> > >, void,
qpid::sys::DispatchHandle&>::invoke(boost::detail::function::function_buffer&,
qpid::sys::DispatchHandle&) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#34 0x00007f16ad1b87f2 in boost::function1<void,
qpid::sys::DispatchHandle&>::op---Type <return> to continue, or q <return>
to quit---
erator()(qpid::sys::DispatchHandle&) const ()
from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#35 0x00007f16ad1b7dfc in
qpid::sys::DispatchHandle::processEvent(qpid::sys::Poller::EventType) ()
from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#36 0x00007f16ad151f66 in qpid::sys::Poller::Event::process() ()
from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#37 0x00007f16ad151264 in qpid::sys::Poller::run() ()
from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#38 0x00007f16ad1b6999 in qpid::sys::Dispatcher::run() ()
from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#39 0x00007f16ad144e6b in qpid::sys::(anonymous
namespace)::runRunnable(void*)
() from /pilot/qpid/qpid-0.28/lib64/libqpidcommon.so.2
#40 0x00000036e58079d1 in start_thread (arg=0x7f162abfd700)
at pthread_create.c:301
#41 0x00000036e54e8b6d in ?? () at ../sysdeps/unix/sysv/linux/makedev.c:33
from /lib64/libc.so.6
#42 0x0000000000000000 in ?? ()