You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "michael j. goulish (JIRA)" <ji...@apache.org> on 2011/03/31 15:56:05 UTC
[jira] [Created] (QPID-3174) broker crash using rdma+sasl
broker crash using rdma+sasl
----------------------------
Key: QPID-3174
URL: https://issues.apache.org/jira/browse/QPID-3174
Project: Qpid
Issue Type: Bug
Components: C++ Broker
Reporter: michael j. goulish
This is sometimes very hard to observe. I was lucky enough to see it three times in the space of 500 iterations yesterday, but now, with same code & scripts & system, I have no repetition after 6000 iterations.
my tree rev number is 1084895
Here is the backtrace -- all 3 have been identical:
#0 0x0000003f8d030265 in raise (sig=<value optimized out>)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x0000003f8d031d10 in abort () at abort.c:88
#2 0x0000003f8d0296e6 in __assert_fail (
assertion=0x2b28f90400aa "completionsNeeded.get() > 0",
file=0x2b28f903fe58 "./qpid/broker/AsyncCompletion.h", line=167,
function=0x2b28f9040440 "void qpid::broker::AsyncCompletion::end(qpid::broker::AsyncCompletion::Callback&)") at assert.c:78
#3 0x00002b28f8fe2935 in qpid::broker::AsyncCompletion::end (this=0x2aaab60c2e50,
cb=<value optimized out>) at ./qpid/broker/AsyncCompletion.h:167
#4 0x00002b28f8fdf3f3 in qpid::broker::SessionState::handleContent (
this=0x2aaabc4157d0, frame=..., id=<value optimized out>)
at qpid/broker/SessionState.cpp:265
#5 0x00002b28f8fdf830 in qpid::broker::SessionState::handleIn (this=0x2aaabc4157d0,
frame=...) at qpid/broker/SessionState.cpp:355
#6 0x00002b28f94799dc in qpid::amqp_0_10::SessionHandler::handleIn (
this=0x2aaab0025680, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
#7 0x00002b28f8f2622b in operator() (this=0x2aaab38cffa0, frame=...)
at ./qpid/framing/Handler.h:42
#8 qpid::broker::Connection::received (this=0x2aaab38cffa0, frame=...)
at qpid/broker/Connection.cpp:164
#9 0x00002b28f8ef8300 in qpid::amqp_0_10::Connection::decode (this=0x2aaabc47d240,
buffer=<value optimized out>, size=<value optimized out>)
at qpid/amqp_0_10/Connection.cpp:58
#10 0x00002b28f94c25e5 in qpid::sys::cyrus::CyrusSecurityLayer::decode (
this=0x2aaabc468480, input=0x2aaab2aa97d0 "", size=118)
at qpid/sys/cyrus/CyrusSecurityLayer.cpp:59
#11 0x00002b28f999c7d8 in qpid::sys::RdmaIOHandler::readbuff (this=0x2aaab002d9e0,
buff=0xffffffffffffffff) at qpid/sys/RdmaIOPlugin.cpp:218
#12 0x00002b28f9bba75a in boost::function2<void, Rdma::AsynchIO&, Rdma::Buffer*, std::allocator<boost::function_base> >::operator() (this=0x0, a0=..., a1=0x6)
here is my broker start script:
-------------- start script -----------------------
#! /bin/bash
export LD_LIBRARY_PATH=$TRUNK/qpid/cpp/src/.libs
QPID_SRC=$TRUNK/qpid/cpp/src
QPIDD=${QPID_SRC}/.libs/qpidd
echo $QPIDD
rm -rf /tmp/mick
mkdir /tmp/mick
$QPIDD \
--no-module-dir \
--load-module ${QPID_SRC}/.libs/rdma.so \
--data-dir /tmp/mick/data_1 \
--auth=yes \
--mgmt-enable=yes \
--port 5813 \
--log-enable info+ \
--log-to-file /tmp/mick/qpidd_1.log \
--log-source yes \
--sasl-config=${QPID_SRC}/tests/sasl_config \
-d
echo "started broker from $QPIDD"
----------- end script ------------------------
and here is my client iterator script;
---------------------- start script ---------------------
#! /bin/bash
rm core.* ~/.qpidd/core*
count=0
while [ $count -lt 10000 ]
do
echo "==========================================="
echo "TEST $count"
echo "==========================================="
sleep 1
core_files=`ls -l core.* ~/.qpidd/core* | wc -l`
echo "core files: ${core_files}"
if [ ${core_files} -gt 0 ]; then
echo "core files found!"
exit 1
else
echo "no core files found."
fi
./qpid-perftest --username zig --password zig --protocol rdma --broker 20.0.40.14 --port 5813 --qt 4 --count 10
count=$(( $count + 1 ))
done
---------------------- end script -----------------------
the sasl config directory that the broker is pointing at was created by the script cpp/src/tests/sasl_test_setup.sh
also, you need to set "ulimit -l 131072" (at least that value) before starting the broker.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] [Commented] (QPID-3174) broker crash using rdma+sasl
Posted by "Gordon Sim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/QPID-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015422#comment-13015422 ]
Gordon Sim commented on QPID-3174:
----------------------------------
I think these fixes should be ported to 0-10 as they are regressions from the flow control change.
> broker crash using rdma+sasl
> ----------------------------
>
> Key: QPID-3174
> URL: https://issues.apache.org/jira/browse/QPID-3174
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Affects Versions: 0.9
> Reporter: michael j. goulish
> Assignee: Ken Giusti
> Fix For: 0.10
>
>
> This is sometimes very hard to observe. I was lucky enough to see it three times in the space of 500 iterations yesterday, but now, with same code & scripts & system, I have no repetition after 6000 iterations.
> my tree rev number is 1084895
> Here is the backtrace -- all 3 have been identical:
> #0 0x0000003f8d030265 in raise (sig=<value optimized out>)
> at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1 0x0000003f8d031d10 in abort () at abort.c:88
> #2 0x0000003f8d0296e6 in __assert_fail (
> assertion=0x2b28f90400aa "completionsNeeded.get() > 0",
> file=0x2b28f903fe58 "./qpid/broker/AsyncCompletion.h", line=167,
> function=0x2b28f9040440 "void qpid::broker::AsyncCompletion::end(qpid::broker::AsyncCompletion::Callback&)") at assert.c:78
> #3 0x00002b28f8fe2935 in qpid::broker::AsyncCompletion::end (this=0x2aaab60c2e50,
> cb=<value optimized out>) at ./qpid/broker/AsyncCompletion.h:167
> #4 0x00002b28f8fdf3f3 in qpid::broker::SessionState::handleContent (
> this=0x2aaabc4157d0, frame=..., id=<value optimized out>)
> at qpid/broker/SessionState.cpp:265
> #5 0x00002b28f8fdf830 in qpid::broker::SessionState::handleIn (this=0x2aaabc4157d0,
> frame=...) at qpid/broker/SessionState.cpp:355
> #6 0x00002b28f94799dc in qpid::amqp_0_10::SessionHandler::handleIn (
> this=0x2aaab0025680, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
> #7 0x00002b28f8f2622b in operator() (this=0x2aaab38cffa0, frame=...)
> at ./qpid/framing/Handler.h:42
> #8 qpid::broker::Connection::received (this=0x2aaab38cffa0, frame=...)
> at qpid/broker/Connection.cpp:164
> #9 0x00002b28f8ef8300 in qpid::amqp_0_10::Connection::decode (this=0x2aaabc47d240,
> buffer=<value optimized out>, size=<value optimized out>)
> at qpid/amqp_0_10/Connection.cpp:58
> #10 0x00002b28f94c25e5 in qpid::sys::cyrus::CyrusSecurityLayer::decode (
> this=0x2aaabc468480, input=0x2aaab2aa97d0 "", size=118)
> at qpid/sys/cyrus/CyrusSecurityLayer.cpp:59
> #11 0x00002b28f999c7d8 in qpid::sys::RdmaIOHandler::readbuff (this=0x2aaab002d9e0,
> buff=0xffffffffffffffff) at qpid/sys/RdmaIOPlugin.cpp:218
> #12 0x00002b28f9bba75a in boost::function2<void, Rdma::AsynchIO&, Rdma::Buffer*, std::allocator<boost::function_base> >::operator() (this=0x0, a0=..., a1=0x6)
> here is my broker start script:
> -------------- start script -----------------------
> #! /bin/bash
> export LD_LIBRARY_PATH=$TRUNK/qpid/cpp/src/.libs
> QPID_SRC=$TRUNK/qpid/cpp/src
> QPIDD=${QPID_SRC}/.libs/qpidd
> echo $QPIDD
> rm -rf /tmp/mick
> mkdir /tmp/mick
> $QPIDD \
> --no-module-dir \
> --load-module ${QPID_SRC}/.libs/rdma.so \
> --data-dir /tmp/mick/data_1 \
> --auth=yes \
> --mgmt-enable=yes \
> --port 5813 \
> --log-enable info+ \
> --log-to-file /tmp/mick/qpidd_1.log \
> --log-source yes \
> --sasl-config=${QPID_SRC}/tests/sasl_config \
> -d
> echo "started broker from $QPIDD"
> ----------- end script ------------------------
> and here is my client iterator script;
> ---------------------- start script ---------------------
> #! /bin/bash
> rm core.* ~/.qpidd/core*
> count=0
> while [ $count -lt 10000 ]
> do
> echo "==========================================="
> echo "TEST $count"
> echo "==========================================="
> sleep 1
> core_files=`ls -l core.* ~/.qpidd/core* | wc -l`
> echo "core files: ${core_files}"
> if [ ${core_files} -gt 0 ]; then
> echo "core files found!"
> exit 1
> else
> echo "no core files found."
> fi
> ./qpid-perftest --username zig --password zig --protocol rdma --broker 20.0.40.14 --port 5813 --qt 4 --count 10
> count=$(( $count + 1 ))
> done
> ---------------------- end script -----------------------
> the sasl config directory that the broker is pointing at was created by the script cpp/src/tests/sasl_test_setup.sh
> also, you need to set "ulimit -l 131072" (at least that value) before starting the broker.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] [Commented] (QPID-3174) broker crash using rdma+sasl
Posted by "Ken Giusti (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/QPID-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014759#comment-13014759 ]
Ken Giusti commented on QPID-3174:
----------------------------------
This appears to be a very subtle race between a session en-queuing a message on a queue at the exact moment a receiver is de-queuing it.
The bottom thread 4 backtrace below shows an inbound message in the process of being enqueued. At the moment the enqueue is completed and the queue lock is dropped, the top thread accesses the message, but the session is being closed down so the top thread attempts to "requeue" the message back. This happens -before- the enqueuing thread 4 can mark the async completion as done.
I believe the fix is to remove the enqueueComplete call from requeue - this is no longer needed as this does not represent an asynchronous receive path.
(gdb) bt
#0 0x00000033d80329a5 in raise () from /lib64/libc.so.6
#1 0x00000033d8034185 in abort () from /lib64/libc.so.6
#2 0x00000033d802b935 in __assert_fail () from /lib64/libc.so.6
#3 0x00007f9cdd1683b3 in invokeCallback (this=0x17b8f50, msg=...) at ./qpid/broker/AsyncCompletion.h:106
#4 finishCompleter (this=0x17b8f50, msg=...) at ./qpid/broker/AsyncCompletion.h:152
#5 enqueueComplete (this=0x17b8f50, msg=...) at ./qpid/broker/PersistableMessage.h:122
#6 qpid::broker::Queue::requeue (this=0x17b8f50, msg=...) at qpid/broker/Queue.cpp:213
#7 0x00007f9cdd193574 in operator() (this=<value optimized out>, requeue=<value optimized out>)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/bits/stl_function.h:587
#8 for_each<std::reverse_iterator<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*> >, std::const_mem\
_fun_ref_t<void, qpid::broker::DeliveryRecord> > (this=<value optimized out>, requeue=<value optimized out>)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/bits/stl_algo.h:4200
#9 qpid::broker::SemanticState::recover (this=<value optimized out>, requeue=<value optimized out>) at qpid/broker/SemanticState.cpp:545
#10 0x00007f9cdd193815 in qpid::broker::SemanticState::closed (this=0x3322e108) at qpid/broker/SemanticState.cpp:95
#11 0x00007f9cdd1ad0bf in qpid::broker::SessionState::~SessionState (this=0x3322df30, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:98
#12 0x00007f9cdd1ad759 in qpid::broker::SessionState::~SessionState (this=0x3322df30, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:106
#13 0x00007f9cdd1a6d8a in ~auto_ptr (this=0x426d39f0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/backward/auto_ptr.h:168
#14 qpid::broker::SessionHandler::handleDetach (this=0x426d39f0) at qpid/broker/SessionHandler.cpp:70
#15 0x00007f9cdccf0eda in qpid::amqp_0_10::SessionHandler::detach (this=0x426d39f0, name="dd6ef6bc-eca9-45be-892b-0d37a2d0aca8")
at qpid/amqp_0_10/SessionHandler.cpp:188
#16 0x00007f9cdccc00d3 in invoke<qpid::framing::AMQP_AllOperations::SessionHandler> (this=0x7f9cda469260, body=<value optimized out>)
at ./qpid/framing/SessionDetachBody.h:63
#17 qpid::framing::AMQP_AllOperations::SessionHandler::Invoker::visit (this=0x7f9cda469260, body=<value optimized out>) at qpid/framing/AllInvoker.cpp:792
#18 0x00007f9cdccef2ac in invoke<qpid::amqp_0_10::SessionHandler> (this=<value optimized out>, m=<value optimized out>) at ./qpid/framing/Invoker.h:67
#19 qpid::amqp_0_10::SessionHandler::invoke (this=<value optimized out>, m=<value optimized out>) at qpid/amqp_0_10/SessionHandler.cpp:67
#20 0x00007f9cdccef91f in qpid::amqp_0_10::SessionHandler::handleIn (this=0x426d39f0, f=...) at qpid/amqp_0_10/SessionHandler.cpp:82
#21 0x00007f9cdd110fa2 in operator() (this=0x4efac50, frame=...) at ./qpid/framing/Handler.h:42
#22 qpid::broker::Connection::received (this=0x4efac50, frame=...) at qpid/broker/Connection.cpp:164
#23 0x00007f9cdd0e97bd in qpid::amqp_0_10::Connection::decode (this=0x2e16bd0, buffer=<value optimized out>, size=<value optimized out>)
at qpid/amqp_0_10/Connection.cpp:58
#24 0x00007f9cdcd1cf4a in qpid::sys::AsynchIOHandler::readbuff (this=0x7f9ccc6cc220, buff=0x7f9c8f0a4820) at qpid/sys/AsynchIOHandler.cpp:135
#25 0x00007f9cdcc64ee2 in operator() (this=0x7f9c35d858f0, h=...) at /usr/include/boost/function/function_template.hpp:1013
#26 qpid::sys::posix::AsynchIO::readable (this=0x7f9c35d858f0, h=...) at qpid/sys/posix/AsynchIO.cpp:428
#27 0x00007f9cdcd214d3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator()(qpid::sys::DispatchHandle&) const ()
from /root/qpid/cpp/src/.libs/libqpidcommon.so.2
#28 0x00007f9cdcd20621 in qpid::sys::DispatchHandle::processEvent (this=0x7f9c35d858f8, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280
#29 0x00007f9cdcc70832 in process (this=0x179cab0) at ./qpid/sys/Poller.h:131
#30 qpid::sys::Poller::run (this=0x179cab0) at qpid/sys/epoll/EpollPoller.cpp:519
#31 0x00007f9cdcc686ca in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#32 0x00000033d84077e1 in start_thread () from /lib64/libpthread.so.0
#33 0x00000033d80e153d in clone () from /lib64/libc.so.6
(gdb) thread 4
[Switching to thread 4 (Thread 14089)]#0 0x00000033d840e1b5 in __lll_unlock_wake () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00000033d840e1b5 in __lll_unlock_wake () from /lib64/libpthread.so.0
#1 0x00000033d840a707 in _L_unlock_657 () from /lib64/libpthread.so.0
#2 0x00000033d840a66f in pthread_mutex_unlock () from /lib64/libpthread.so.0
#3 0x00007f9cdd0eb1f6 in qpid::sys::Mutex::unlock (this=<value optimized out>) at ../include/qpid/sys/posix/Mutex.h:120
#4 0x00007f9cdd169211 in ~ScopedLock (this=0x17b8f50, msg=<value optimized out>, isRecovery=false) at ../include/qpid/sys/Mutex.h:34
#5 qpid::broker::Queue::push (this=0x17b8f50, msg=<value optimized out>, isRecovery=false) at qpid/broker/Queue.cpp:528
#6 0x00007f9cdd16a3ff in qpid::broker::Queue::deliver (this=0x17b8f50, msg=...) at qpid/broker/Queue.cpp:168
#7 0x00007f9cdd119c1f in qpid::broker::DeliverableMessage::deliverTo (this=0x7fff2d932520, queue=<value optimized out>) at qpid/broker/DeliverableMessage.cpp:33
#8 0x00007f9cdd12dd5a in qpid::broker::Exchange::doRoute (this=0x17b3170, msg=..., b=...) at qpid/broker/Exchange.cpp:84
#9 0x00007f9cdd11da8e in qpid::broker::DirectExchange::route (this=0x17b3110, msg=..., routingKey=<value optimized out>) at qpid/broker/DirectExchange.cpp:160
#10 0x00007f9cdd18d321 in qpid::broker::SemanticState::route (this=0x7f9bfc1ab8b8, msg=..., strategy=...) at qpid/broker/SemanticState.cpp:486
#11 0x00007f9cdd18e311 in qpid::broker::SemanticState::handle (this=0x7f9bfc1ab8b8, msg=...) at qpid/broker/SemanticState.cpp:440
#12 0x00007f9cdd1ac717 in qpid::broker::SessionState::handleContent (this=0x7f9bfc1ab6e0, frame=..., id=<value optimized out>) at qpid/broker/SessionState.cpp:262
#13 0x00007f9cdd1acec7 in qpid::broker::SessionState::handleIn (this=0x7f9bfc1ab6e0, frame=...) at qpid/broker/SessionState.cpp:355
#14 0x00007f9cdccef8d1 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x7f9bfc1ab500, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
#15 0x00007f9cdd110fa2 in operator() (this=0x7f9badfb38e0, frame=...) at ./qpid/framing/Handler.h:42
#16 qpid::broker::Connection::received (this=0x7f9badfb38e0, frame=...) at qpid/broker/Connection.cpp:164
#17 0x00007f9cdd0e97bd in qpid::amqp_0_10::Connection::decode (this=0x7f9badfb5720, buffer=<value optimized out>, size=<value optimized out>)
at qpid/amqp_0_10/Connection.cpp:58
#18 0x00007f9cdcd1cf4a in qpid::sys::AsynchIOHandler::readbuff (this=0x7f9badfb3e70, buff=0x7f9badfb33d0) at qpid/sys/AsynchIOHandler.cpp:135
#19 0x00007f9cdcc64ee2 in operator() (this=0x7f9badfb4c10, h=...) at /usr/include/boost/function/function_template.hpp:1013
#20 qpid::sys::posix::AsynchIO::readable (this=0x7f9badfb4c10, h=...) at qpid/sys/posix/AsynchIO.cpp:428
#21 0x00007f9cdcd214d3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator()(qpid::sys::DispatchHandle&) const ()
from /root/qpid/cpp/src/.libs/libqpidcommon.so.2
#22 0x00007f9cdcd20621 in qpid::sys::DispatchHandle::processEvent (this=0x7f9badfb4c18, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280
#23 0x00007f9cdcc70832 in process (this=0x179cab0) at ./qpid/sys/Poller.h:131
#24 qpid::sys::Poller::run (this=0x179cab0) at qpid/sys/epoll/EpollPoller.cpp:519
#25 0x00007f9cdd0fdd82 in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:385
#26 0x000000000040d82d in QpiddBroker::execute (this=<value optimized out>, options=0x17977b0) at posix/QpiddBroker.cpp:187
#27 0x000000000040a1e2 in main (argc=13, argv=0x7fff2d934de8) at qpidd.cpp:80
(gdb)
> broker crash using rdma+sasl
> ----------------------------
>
> Key: QPID-3174
> URL: https://issues.apache.org/jira/browse/QPID-3174
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Reporter: michael j. goulish
> Assignee: Ken Giusti
>
> This is sometimes very hard to observe. I was lucky enough to see it three times in the space of 500 iterations yesterday, but now, with same code & scripts & system, I have no repetition after 6000 iterations.
> my tree rev number is 1084895
> Here is the backtrace -- all 3 have been identical:
> #0 0x0000003f8d030265 in raise (sig=<value optimized out>)
> at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1 0x0000003f8d031d10 in abort () at abort.c:88
> #2 0x0000003f8d0296e6 in __assert_fail (
> assertion=0x2b28f90400aa "completionsNeeded.get() > 0",
> file=0x2b28f903fe58 "./qpid/broker/AsyncCompletion.h", line=167,
> function=0x2b28f9040440 "void qpid::broker::AsyncCompletion::end(qpid::broker::AsyncCompletion::Callback&)") at assert.c:78
> #3 0x00002b28f8fe2935 in qpid::broker::AsyncCompletion::end (this=0x2aaab60c2e50,
> cb=<value optimized out>) at ./qpid/broker/AsyncCompletion.h:167
> #4 0x00002b28f8fdf3f3 in qpid::broker::SessionState::handleContent (
> this=0x2aaabc4157d0, frame=..., id=<value optimized out>)
> at qpid/broker/SessionState.cpp:265
> #5 0x00002b28f8fdf830 in qpid::broker::SessionState::handleIn (this=0x2aaabc4157d0,
> frame=...) at qpid/broker/SessionState.cpp:355
> #6 0x00002b28f94799dc in qpid::amqp_0_10::SessionHandler::handleIn (
> this=0x2aaab0025680, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
> #7 0x00002b28f8f2622b in operator() (this=0x2aaab38cffa0, frame=...)
> at ./qpid/framing/Handler.h:42
> #8 qpid::broker::Connection::received (this=0x2aaab38cffa0, frame=...)
> at qpid/broker/Connection.cpp:164
> #9 0x00002b28f8ef8300 in qpid::amqp_0_10::Connection::decode (this=0x2aaabc47d240,
> buffer=<value optimized out>, size=<value optimized out>)
> at qpid/amqp_0_10/Connection.cpp:58
> #10 0x00002b28f94c25e5 in qpid::sys::cyrus::CyrusSecurityLayer::decode (
> this=0x2aaabc468480, input=0x2aaab2aa97d0 "", size=118)
> at qpid/sys/cyrus/CyrusSecurityLayer.cpp:59
> #11 0x00002b28f999c7d8 in qpid::sys::RdmaIOHandler::readbuff (this=0x2aaab002d9e0,
> buff=0xffffffffffffffff) at qpid/sys/RdmaIOPlugin.cpp:218
> #12 0x00002b28f9bba75a in boost::function2<void, Rdma::AsynchIO&, Rdma::Buffer*, std::allocator<boost::function_base> >::operator() (this=0x0, a0=..., a1=0x6)
> here is my broker start script:
> -------------- start script -----------------------
> #! /bin/bash
> export LD_LIBRARY_PATH=$TRUNK/qpid/cpp/src/.libs
> QPID_SRC=$TRUNK/qpid/cpp/src
> QPIDD=${QPID_SRC}/.libs/qpidd
> echo $QPIDD
> rm -rf /tmp/mick
> mkdir /tmp/mick
> $QPIDD \
> --no-module-dir \
> --load-module ${QPID_SRC}/.libs/rdma.so \
> --data-dir /tmp/mick/data_1 \
> --auth=yes \
> --mgmt-enable=yes \
> --port 5813 \
> --log-enable info+ \
> --log-to-file /tmp/mick/qpidd_1.log \
> --log-source yes \
> --sasl-config=${QPID_SRC}/tests/sasl_config \
> -d
> echo "started broker from $QPIDD"
> ----------- end script ------------------------
> and here is my client iterator script;
> ---------------------- start script ---------------------
> #! /bin/bash
> rm core.* ~/.qpidd/core*
> count=0
> while [ $count -lt 10000 ]
> do
> echo "==========================================="
> echo "TEST $count"
> echo "==========================================="
> sleep 1
> core_files=`ls -l core.* ~/.qpidd/core* | wc -l`
> echo "core files: ${core_files}"
> if [ ${core_files} -gt 0 ]; then
> echo "core files found!"
> exit 1
> else
> echo "no core files found."
> fi
> ./qpid-perftest --username zig --password zig --protocol rdma --broker 20.0.40.14 --port 5813 --qt 4 --count 10
> count=$(( $count + 1 ))
> done
> ---------------------- end script -----------------------
> the sasl config directory that the broker is pointing at was created by the script cpp/src/tests/sasl_test_setup.sh
> also, you need to set "ulimit -l 131072" (at least that value) before starting the broker.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] [Resolved] (QPID-3174) broker crash using rdma+sasl
Posted by "Gordon Sim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/QPID-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gordon Sim resolved QPID-3174.
------------------------------
Resolution: Fixed
> broker crash using rdma+sasl
> ----------------------------
>
> Key: QPID-3174
> URL: https://issues.apache.org/jira/browse/QPID-3174
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Affects Versions: 0.9
> Reporter: michael j. goulish
> Assignee: Ken Giusti
> Fix For: 0.10
>
>
> This is sometimes very hard to observe. I was lucky enough to see it three times in the space of 500 iterations yesterday, but now, with same code & scripts & system, I have no repetition after 6000 iterations.
> my tree rev number is 1084895
> Here is the backtrace -- all 3 have been identical:
> #0 0x0000003f8d030265 in raise (sig=<value optimized out>)
> at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1 0x0000003f8d031d10 in abort () at abort.c:88
> #2 0x0000003f8d0296e6 in __assert_fail (
> assertion=0x2b28f90400aa "completionsNeeded.get() > 0",
> file=0x2b28f903fe58 "./qpid/broker/AsyncCompletion.h", line=167,
> function=0x2b28f9040440 "void qpid::broker::AsyncCompletion::end(qpid::broker::AsyncCompletion::Callback&)") at assert.c:78
> #3 0x00002b28f8fe2935 in qpid::broker::AsyncCompletion::end (this=0x2aaab60c2e50,
> cb=<value optimized out>) at ./qpid/broker/AsyncCompletion.h:167
> #4 0x00002b28f8fdf3f3 in qpid::broker::SessionState::handleContent (
> this=0x2aaabc4157d0, frame=..., id=<value optimized out>)
> at qpid/broker/SessionState.cpp:265
> #5 0x00002b28f8fdf830 in qpid::broker::SessionState::handleIn (this=0x2aaabc4157d0,
> frame=...) at qpid/broker/SessionState.cpp:355
> #6 0x00002b28f94799dc in qpid::amqp_0_10::SessionHandler::handleIn (
> this=0x2aaab0025680, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
> #7 0x00002b28f8f2622b in operator() (this=0x2aaab38cffa0, frame=...)
> at ./qpid/framing/Handler.h:42
> #8 qpid::broker::Connection::received (this=0x2aaab38cffa0, frame=...)
> at qpid/broker/Connection.cpp:164
> #9 0x00002b28f8ef8300 in qpid::amqp_0_10::Connection::decode (this=0x2aaabc47d240,
> buffer=<value optimized out>, size=<value optimized out>)
> at qpid/amqp_0_10/Connection.cpp:58
> #10 0x00002b28f94c25e5 in qpid::sys::cyrus::CyrusSecurityLayer::decode (
> this=0x2aaabc468480, input=0x2aaab2aa97d0 "", size=118)
> at qpid/sys/cyrus/CyrusSecurityLayer.cpp:59
> #11 0x00002b28f999c7d8 in qpid::sys::RdmaIOHandler::readbuff (this=0x2aaab002d9e0,
> buff=0xffffffffffffffff) at qpid/sys/RdmaIOPlugin.cpp:218
> #12 0x00002b28f9bba75a in boost::function2<void, Rdma::AsynchIO&, Rdma::Buffer*, std::allocator<boost::function_base> >::operator() (this=0x0, a0=..., a1=0x6)
> here is my broker start script:
> -------------- start script -----------------------
> #! /bin/bash
> export LD_LIBRARY_PATH=$TRUNK/qpid/cpp/src/.libs
> QPID_SRC=$TRUNK/qpid/cpp/src
> QPIDD=${QPID_SRC}/.libs/qpidd
> echo $QPIDD
> rm -rf /tmp/mick
> mkdir /tmp/mick
> $QPIDD \
> --no-module-dir \
> --load-module ${QPID_SRC}/.libs/rdma.so \
> --data-dir /tmp/mick/data_1 \
> --auth=yes \
> --mgmt-enable=yes \
> --port 5813 \
> --log-enable info+ \
> --log-to-file /tmp/mick/qpidd_1.log \
> --log-source yes \
> --sasl-config=${QPID_SRC}/tests/sasl_config \
> -d
> echo "started broker from $QPIDD"
> ----------- end script ------------------------
> and here is my client iterator script;
> ---------------------- start script ---------------------
> #! /bin/bash
> rm core.* ~/.qpidd/core*
> count=0
> while [ $count -lt 10000 ]
> do
> echo "==========================================="
> echo "TEST $count"
> echo "==========================================="
> sleep 1
> core_files=`ls -l core.* ~/.qpidd/core* | wc -l`
> echo "core files: ${core_files}"
> if [ ${core_files} -gt 0 ]; then
> echo "core files found!"
> exit 1
> else
> echo "no core files found."
> fi
> ./qpid-perftest --username zig --password zig --protocol rdma --broker 20.0.40.14 --port 5813 --qt 4 --count 10
> count=$(( $count + 1 ))
> done
> ---------------------- end script -----------------------
> the sasl config directory that the broker is pointing at was created by the script cpp/src/tests/sasl_test_setup.sh
> also, you need to set "ulimit -l 131072" (at least that value) before starting the broker.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] [Commented] (QPID-3174) broker crash using rdma+sasl
Posted by "Justin Ross (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/QPID-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015428#comment-13015428 ]
Justin Ross commented on QPID-3174:
-----------------------------------
Approved for 0.10. Reviewed by Gordon Sim.
> broker crash using rdma+sasl
> ----------------------------
>
> Key: QPID-3174
> URL: https://issues.apache.org/jira/browse/QPID-3174
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Affects Versions: 0.9
> Reporter: michael j. goulish
> Assignee: Ken Giusti
> Fix For: 0.10
>
>
> This is sometimes very hard to observe. I was lucky enough to see it three times in the space of 500 iterations yesterday, but now, with same code & scripts & system, I have no repetition after 6000 iterations.
> my tree rev number is 1084895
> Here is the backtrace -- all 3 have been identical:
> #0 0x0000003f8d030265 in raise (sig=<value optimized out>)
> at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1 0x0000003f8d031d10 in abort () at abort.c:88
> #2 0x0000003f8d0296e6 in __assert_fail (
> assertion=0x2b28f90400aa "completionsNeeded.get() > 0",
> file=0x2b28f903fe58 "./qpid/broker/AsyncCompletion.h", line=167,
> function=0x2b28f9040440 "void qpid::broker::AsyncCompletion::end(qpid::broker::AsyncCompletion::Callback&)") at assert.c:78
> #3 0x00002b28f8fe2935 in qpid::broker::AsyncCompletion::end (this=0x2aaab60c2e50,
> cb=<value optimized out>) at ./qpid/broker/AsyncCompletion.h:167
> #4 0x00002b28f8fdf3f3 in qpid::broker::SessionState::handleContent (
> this=0x2aaabc4157d0, frame=..., id=<value optimized out>)
> at qpid/broker/SessionState.cpp:265
> #5 0x00002b28f8fdf830 in qpid::broker::SessionState::handleIn (this=0x2aaabc4157d0,
> frame=...) at qpid/broker/SessionState.cpp:355
> #6 0x00002b28f94799dc in qpid::amqp_0_10::SessionHandler::handleIn (
> this=0x2aaab0025680, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
> #7 0x00002b28f8f2622b in operator() (this=0x2aaab38cffa0, frame=...)
> at ./qpid/framing/Handler.h:42
> #8 qpid::broker::Connection::received (this=0x2aaab38cffa0, frame=...)
> at qpid/broker/Connection.cpp:164
> #9 0x00002b28f8ef8300 in qpid::amqp_0_10::Connection::decode (this=0x2aaabc47d240,
> buffer=<value optimized out>, size=<value optimized out>)
> at qpid/amqp_0_10/Connection.cpp:58
> #10 0x00002b28f94c25e5 in qpid::sys::cyrus::CyrusSecurityLayer::decode (
> this=0x2aaabc468480, input=0x2aaab2aa97d0 "", size=118)
> at qpid/sys/cyrus/CyrusSecurityLayer.cpp:59
> #11 0x00002b28f999c7d8 in qpid::sys::RdmaIOHandler::readbuff (this=0x2aaab002d9e0,
> buff=0xffffffffffffffff) at qpid/sys/RdmaIOPlugin.cpp:218
> #12 0x00002b28f9bba75a in boost::function2<void, Rdma::AsynchIO&, Rdma::Buffer*, std::allocator<boost::function_base> >::operator() (this=0x0, a0=..., a1=0x6)
> here is my broker start script:
> -------------- start script -----------------------
> #! /bin/bash
> export LD_LIBRARY_PATH=$TRUNK/qpid/cpp/src/.libs
> QPID_SRC=$TRUNK/qpid/cpp/src
> QPIDD=${QPID_SRC}/.libs/qpidd
> echo $QPIDD
> rm -rf /tmp/mick
> mkdir /tmp/mick
> $QPIDD \
> --no-module-dir \
> --load-module ${QPID_SRC}/.libs/rdma.so \
> --data-dir /tmp/mick/data_1 \
> --auth=yes \
> --mgmt-enable=yes \
> --port 5813 \
> --log-enable info+ \
> --log-to-file /tmp/mick/qpidd_1.log \
> --log-source yes \
> --sasl-config=${QPID_SRC}/tests/sasl_config \
> -d
> echo "started broker from $QPIDD"
> ----------- end script ------------------------
> and here is my client iterator script;
> ---------------------- start script ---------------------
> #! /bin/bash
> rm core.* ~/.qpidd/core*
> count=0
> while [ $count -lt 10000 ]
> do
> echo "==========================================="
> echo "TEST $count"
> echo "==========================================="
> sleep 1
> core_files=`ls -l core.* ~/.qpidd/core* | wc -l`
> echo "core files: ${core_files}"
> if [ ${core_files} -gt 0 ]; then
> echo "core files found!"
> exit 1
> else
> echo "no core files found."
> fi
> ./qpid-perftest --username zig --password zig --protocol rdma --broker 20.0.40.14 --port 5813 --qt 4 --count 10
> count=$(( $count + 1 ))
> done
> ---------------------- end script -----------------------
> the sasl config directory that the broker is pointing at was created by the script cpp/src/tests/sasl_test_setup.sh
> also, you need to set "ulimit -l 131072" (at least that value) before starting the broker.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] [Updated] (QPID-3174) broker crash using rdma+sasl
Posted by "Ken Giusti (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/QPID-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ken Giusti updated QPID-3174:
-----------------------------
Affects Version/s: 0.9
Fix Version/s: 0.10
> broker crash using rdma+sasl
> ----------------------------
>
> Key: QPID-3174
> URL: https://issues.apache.org/jira/browse/QPID-3174
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Affects Versions: 0.9
> Reporter: michael j. goulish
> Assignee: Ken Giusti
> Fix For: 0.10
>
>
> This is sometimes very hard to observe. I was lucky enough to see it three times in the space of 500 iterations yesterday, but now, with same code & scripts & system, I have no repetition after 6000 iterations.
> my tree rev number is 1084895
> Here is the backtrace -- all 3 have been identical:
> #0 0x0000003f8d030265 in raise (sig=<value optimized out>)
> at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1 0x0000003f8d031d10 in abort () at abort.c:88
> #2 0x0000003f8d0296e6 in __assert_fail (
> assertion=0x2b28f90400aa "completionsNeeded.get() > 0",
> file=0x2b28f903fe58 "./qpid/broker/AsyncCompletion.h", line=167,
> function=0x2b28f9040440 "void qpid::broker::AsyncCompletion::end(qpid::broker::AsyncCompletion::Callback&)") at assert.c:78
> #3 0x00002b28f8fe2935 in qpid::broker::AsyncCompletion::end (this=0x2aaab60c2e50,
> cb=<value optimized out>) at ./qpid/broker/AsyncCompletion.h:167
> #4 0x00002b28f8fdf3f3 in qpid::broker::SessionState::handleContent (
> this=0x2aaabc4157d0, frame=..., id=<value optimized out>)
> at qpid/broker/SessionState.cpp:265
> #5 0x00002b28f8fdf830 in qpid::broker::SessionState::handleIn (this=0x2aaabc4157d0,
> frame=...) at qpid/broker/SessionState.cpp:355
> #6 0x00002b28f94799dc in qpid::amqp_0_10::SessionHandler::handleIn (
> this=0x2aaab0025680, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
> #7 0x00002b28f8f2622b in operator() (this=0x2aaab38cffa0, frame=...)
> at ./qpid/framing/Handler.h:42
> #8 qpid::broker::Connection::received (this=0x2aaab38cffa0, frame=...)
> at qpid/broker/Connection.cpp:164
> #9 0x00002b28f8ef8300 in qpid::amqp_0_10::Connection::decode (this=0x2aaabc47d240,
> buffer=<value optimized out>, size=<value optimized out>)
> at qpid/amqp_0_10/Connection.cpp:58
> #10 0x00002b28f94c25e5 in qpid::sys::cyrus::CyrusSecurityLayer::decode (
> this=0x2aaabc468480, input=0x2aaab2aa97d0 "", size=118)
> at qpid/sys/cyrus/CyrusSecurityLayer.cpp:59
> #11 0x00002b28f999c7d8 in qpid::sys::RdmaIOHandler::readbuff (this=0x2aaab002d9e0,
> buff=0xffffffffffffffff) at qpid/sys/RdmaIOPlugin.cpp:218
> #12 0x00002b28f9bba75a in boost::function2<void, Rdma::AsynchIO&, Rdma::Buffer*, std::allocator<boost::function_base> >::operator() (this=0x0, a0=..., a1=0x6)
> here is my broker start script:
> -------------- start script -----------------------
> #! /bin/bash
> export LD_LIBRARY_PATH=$TRUNK/qpid/cpp/src/.libs
> QPID_SRC=$TRUNK/qpid/cpp/src
> QPIDD=${QPID_SRC}/.libs/qpidd
> echo $QPIDD
> rm -rf /tmp/mick
> mkdir /tmp/mick
> $QPIDD \
> --no-module-dir \
> --load-module ${QPID_SRC}/.libs/rdma.so \
> --data-dir /tmp/mick/data_1 \
> --auth=yes \
> --mgmt-enable=yes \
> --port 5813 \
> --log-enable info+ \
> --log-to-file /tmp/mick/qpidd_1.log \
> --log-source yes \
> --sasl-config=${QPID_SRC}/tests/sasl_config \
> -d
> echo "started broker from $QPIDD"
> ----------- end script ------------------------
> and here is my client iterator script;
> ---------------------- start script ---------------------
> #! /bin/bash
> rm core.* ~/.qpidd/core*
> count=0
> while [ $count -lt 10000 ]
> do
> echo "==========================================="
> echo "TEST $count"
> echo "==========================================="
> sleep 1
> core_files=`ls -l core.* ~/.qpidd/core* | wc -l`
> echo "core files: ${core_files}"
> if [ ${core_files} -gt 0 ]; then
> echo "core files found!"
> exit 1
> else
> echo "no core files found."
> fi
> ./qpid-perftest --username zig --password zig --protocol rdma --broker 20.0.40.14 --port 5813 --qt 4 --count 10
> count=$(( $count + 1 ))
> done
> ---------------------- end script -----------------------
> the sasl config directory that the broker is pointing at was created by the script cpp/src/tests/sasl_test_setup.sh
> also, you need to set "ulimit -l 131072" (at least that value) before starting the broker.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] [Assigned] (QPID-3174) broker crash using rdma+sasl
Posted by "Ken Giusti (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/QPID-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ken Giusti reassigned QPID-3174:
--------------------------------
Assignee: Ken Giusti
> broker crash using rdma+sasl
> ----------------------------
>
> Key: QPID-3174
> URL: https://issues.apache.org/jira/browse/QPID-3174
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Reporter: michael j. goulish
> Assignee: Ken Giusti
>
> This is sometimes very hard to observe. I was lucky enough to see it three times in the space of 500 iterations yesterday, but now, with same code & scripts & system, I have no repetition after 6000 iterations.
> my tree rev number is 1084895
> Here is the backtrace -- all 3 have been identical:
> #0 0x0000003f8d030265 in raise (sig=<value optimized out>)
> at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1 0x0000003f8d031d10 in abort () at abort.c:88
> #2 0x0000003f8d0296e6 in __assert_fail (
> assertion=0x2b28f90400aa "completionsNeeded.get() > 0",
> file=0x2b28f903fe58 "./qpid/broker/AsyncCompletion.h", line=167,
> function=0x2b28f9040440 "void qpid::broker::AsyncCompletion::end(qpid::broker::AsyncCompletion::Callback&)") at assert.c:78
> #3 0x00002b28f8fe2935 in qpid::broker::AsyncCompletion::end (this=0x2aaab60c2e50,
> cb=<value optimized out>) at ./qpid/broker/AsyncCompletion.h:167
> #4 0x00002b28f8fdf3f3 in qpid::broker::SessionState::handleContent (
> this=0x2aaabc4157d0, frame=..., id=<value optimized out>)
> at qpid/broker/SessionState.cpp:265
> #5 0x00002b28f8fdf830 in qpid::broker::SessionState::handleIn (this=0x2aaabc4157d0,
> frame=...) at qpid/broker/SessionState.cpp:355
> #6 0x00002b28f94799dc in qpid::amqp_0_10::SessionHandler::handleIn (
> this=0x2aaab0025680, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
> #7 0x00002b28f8f2622b in operator() (this=0x2aaab38cffa0, frame=...)
> at ./qpid/framing/Handler.h:42
> #8 qpid::broker::Connection::received (this=0x2aaab38cffa0, frame=...)
> at qpid/broker/Connection.cpp:164
> #9 0x00002b28f8ef8300 in qpid::amqp_0_10::Connection::decode (this=0x2aaabc47d240,
> buffer=<value optimized out>, size=<value optimized out>)
> at qpid/amqp_0_10/Connection.cpp:58
> #10 0x00002b28f94c25e5 in qpid::sys::cyrus::CyrusSecurityLayer::decode (
> this=0x2aaabc468480, input=0x2aaab2aa97d0 "", size=118)
> at qpid/sys/cyrus/CyrusSecurityLayer.cpp:59
> #11 0x00002b28f999c7d8 in qpid::sys::RdmaIOHandler::readbuff (this=0x2aaab002d9e0,
> buff=0xffffffffffffffff) at qpid/sys/RdmaIOPlugin.cpp:218
> #12 0x00002b28f9bba75a in boost::function2<void, Rdma::AsynchIO&, Rdma::Buffer*, std::allocator<boost::function_base> >::operator() (this=0x0, a0=..., a1=0x6)
> here is my broker start script:
> -------------- start script -----------------------
> #! /bin/bash
> export LD_LIBRARY_PATH=$TRUNK/qpid/cpp/src/.libs
> QPID_SRC=$TRUNK/qpid/cpp/src
> QPIDD=${QPID_SRC}/.libs/qpidd
> echo $QPIDD
> rm -rf /tmp/mick
> mkdir /tmp/mick
> $QPIDD \
> --no-module-dir \
> --load-module ${QPID_SRC}/.libs/rdma.so \
> --data-dir /tmp/mick/data_1 \
> --auth=yes \
> --mgmt-enable=yes \
> --port 5813 \
> --log-enable info+ \
> --log-to-file /tmp/mick/qpidd_1.log \
> --log-source yes \
> --sasl-config=${QPID_SRC}/tests/sasl_config \
> -d
> echo "started broker from $QPIDD"
> ----------- end script ------------------------
> and here is my client iterator script;
> ---------------------- start script ---------------------
> #! /bin/bash
> rm core.* ~/.qpidd/core*
> count=0
> while [ $count -lt 10000 ]
> do
> echo "==========================================="
> echo "TEST $count"
> echo "==========================================="
> sleep 1
> core_files=`ls -l core.* ~/.qpidd/core* | wc -l`
> echo "core files: ${core_files}"
> if [ ${core_files} -gt 0 ]; then
> echo "core files found!"
> exit 1
> else
> echo "no core files found."
> fi
> ./qpid-perftest --username zig --password zig --protocol rdma --broker 20.0.40.14 --port 5813 --qt 4 --count 10
> count=$(( $count + 1 ))
> done
> ---------------------- end script -----------------------
> the sasl config directory that the broker is pointing at was created by the script cpp/src/tests/sasl_test_setup.sh
> also, you need to set "ulimit -l 131072" (at least that value) before starting the broker.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] [Commented] (QPID-3174) broker crash using rdma+sasl
Posted by "Ken Giusti (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/QPID-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014548#comment-13014548 ]
Ken Giusti commented on QPID-3174:
----------------------------------
Most certainly caused by my implementation of https://issues.apache.org/jira/browse/QPID-2921. Investigating.
> broker crash using rdma+sasl
> ----------------------------
>
> Key: QPID-3174
> URL: https://issues.apache.org/jira/browse/QPID-3174
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Reporter: michael j. goulish
> Assignee: Ken Giusti
>
> This is sometimes very hard to observe. I was lucky enough to see it three times in the space of 500 iterations yesterday, but now, with same code & scripts & system, I have no repetition after 6000 iterations.
> my tree rev number is 1084895
> Here is the backtrace -- all 3 have been identical:
> #0 0x0000003f8d030265 in raise (sig=<value optimized out>)
> at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1 0x0000003f8d031d10 in abort () at abort.c:88
> #2 0x0000003f8d0296e6 in __assert_fail (
> assertion=0x2b28f90400aa "completionsNeeded.get() > 0",
> file=0x2b28f903fe58 "./qpid/broker/AsyncCompletion.h", line=167,
> function=0x2b28f9040440 "void qpid::broker::AsyncCompletion::end(qpid::broker::AsyncCompletion::Callback&)") at assert.c:78
> #3 0x00002b28f8fe2935 in qpid::broker::AsyncCompletion::end (this=0x2aaab60c2e50,
> cb=<value optimized out>) at ./qpid/broker/AsyncCompletion.h:167
> #4 0x00002b28f8fdf3f3 in qpid::broker::SessionState::handleContent (
> this=0x2aaabc4157d0, frame=..., id=<value optimized out>)
> at qpid/broker/SessionState.cpp:265
> #5 0x00002b28f8fdf830 in qpid::broker::SessionState::handleIn (this=0x2aaabc4157d0,
> frame=...) at qpid/broker/SessionState.cpp:355
> #6 0x00002b28f94799dc in qpid::amqp_0_10::SessionHandler::handleIn (
> this=0x2aaab0025680, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
> #7 0x00002b28f8f2622b in operator() (this=0x2aaab38cffa0, frame=...)
> at ./qpid/framing/Handler.h:42
> #8 qpid::broker::Connection::received (this=0x2aaab38cffa0, frame=...)
> at qpid/broker/Connection.cpp:164
> #9 0x00002b28f8ef8300 in qpid::amqp_0_10::Connection::decode (this=0x2aaabc47d240,
> buffer=<value optimized out>, size=<value optimized out>)
> at qpid/amqp_0_10/Connection.cpp:58
> #10 0x00002b28f94c25e5 in qpid::sys::cyrus::CyrusSecurityLayer::decode (
> this=0x2aaabc468480, input=0x2aaab2aa97d0 "", size=118)
> at qpid/sys/cyrus/CyrusSecurityLayer.cpp:59
> #11 0x00002b28f999c7d8 in qpid::sys::RdmaIOHandler::readbuff (this=0x2aaab002d9e0,
> buff=0xffffffffffffffff) at qpid/sys/RdmaIOPlugin.cpp:218
> #12 0x00002b28f9bba75a in boost::function2<void, Rdma::AsynchIO&, Rdma::Buffer*, std::allocator<boost::function_base> >::operator() (this=0x0, a0=..., a1=0x6)
> here is my broker start script:
> -------------- start script -----------------------
> #! /bin/bash
> export LD_LIBRARY_PATH=$TRUNK/qpid/cpp/src/.libs
> QPID_SRC=$TRUNK/qpid/cpp/src
> QPIDD=${QPID_SRC}/.libs/qpidd
> echo $QPIDD
> rm -rf /tmp/mick
> mkdir /tmp/mick
> $QPIDD \
> --no-module-dir \
> --load-module ${QPID_SRC}/.libs/rdma.so \
> --data-dir /tmp/mick/data_1 \
> --auth=yes \
> --mgmt-enable=yes \
> --port 5813 \
> --log-enable info+ \
> --log-to-file /tmp/mick/qpidd_1.log \
> --log-source yes \
> --sasl-config=${QPID_SRC}/tests/sasl_config \
> -d
> echo "started broker from $QPIDD"
> ----------- end script ------------------------
> and here is my client iterator script;
> ---------------------- start script ---------------------
> #! /bin/bash
> rm core.* ~/.qpidd/core*
> count=0
> while [ $count -lt 10000 ]
> do
> echo "==========================================="
> echo "TEST $count"
> echo "==========================================="
> sleep 1
> core_files=`ls -l core.* ~/.qpidd/core* | wc -l`
> echo "core files: ${core_files}"
> if [ ${core_files} -gt 0 ]; then
> echo "core files found!"
> exit 1
> else
> echo "no core files found."
> fi
> ./qpid-perftest --username zig --password zig --protocol rdma --broker 20.0.40.14 --port 5813 --qt 4 --count 10
> count=$(( $count + 1 ))
> done
> ---------------------- end script -----------------------
> the sasl config directory that the broker is pointing at was created by the script cpp/src/tests/sasl_test_setup.sh
> also, you need to set "ulimit -l 131072" (at least that value) before starting the broker.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org