[jira] [Created] (AMQCPP-753) Deadlock when connection fails before ack can be delivered

["Katherine Pully (Jira)" <ji...@apache.org>]

Katherine Pully created AMQCPP-753:
--------------------------------------

             Summary: Deadlock when connection fails before ack can be delivered
                 Key: AMQCPP-753
                 URL: https://issues.apache.org/jira/browse/AMQCPP-753
             Project: ActiveMQ C++ Client
          Issue Type: Bug
          Components: Decaf
    Affects Versions: 3.9.5
         Environment: Unix/Linux
            Reporter: Katherine Pully
            Assignee: Timothy A. Bish
         Attachments: recreated_hang.txt

When a connection fails before a message acknowledgement can be delivered, only decaf exceptions are caught. However, the check for a failed connection ultimately results in a CMS exception. As a result, this exception is not handled and the consumer read lock does not get released. This will result in a deadlock when the session tries to acquire the consumer write lock (for example, when cleaning up the connection).

I have attached a stack trace from such a deadlock, which occurs when the connection is cleaned up. The relevant portion (edited for brevity and clarity, though the attached is the original), is:
{code:java}
0  decaf::util::concurrent::ExecutorKernel::Worker::run() ThreadPoolExecutor.cpp:184
1  decaf::util::concurrent::ExecutorKernel::runWorker (decaf::util::concurrent::ExecutorKernel::Worker*) ThreadPoolExecutor.cpp:738
2  activemq::core::OnExceptionRunnable::run() ActiveMQConnection.cpp:439
3  activemq::core::ActiveMQConnection::cleanup() ActiveMQConnection.cpp:839
4  activemq::core::kernels::ActiveMQSessionKernel::dispose() ActiveMQSessionKernel.cpp:371
5  decaf::util::concurrent::locks::AbstractQueuedSynchronizer::acquire(int) AbstractQueuedSynchronizer.cpp:1565
6  decaf::util::concurrent::locks::SynchronizerState::acquireQueued((anonymous namespace)::Node*, int) AbstractQueuedSynchronizer.cpp:711
7  decaf::util::concurrent::locks::LockSupport::park() LockSupport.cpp:54
8  decaf::internal::util::concurrent::Threading::park(decaf::lang::Thread*) Threading.cpp:1345
9  decaf::internal::util::concurrent::PlatformThread::interruptibleWaitOnCondition(_opaque_pthread_cond_t*, _opaque_pthread_mutex_t*, decaf::internal::util::concurrent::CompletionCondition&) PlatformThread.cpp:210
10 _pthread_cond_wait
11 __psynch_cvwait{code}

The issue can be produced by using a client-acknowledge strategy and adding a substantial (10+ seconds) call to sleep before acknowledging the message, and then breaking the connection.

The exception is originally thrown by [ActiveMQConnection::checkClosedOrFailed|[https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/ActiveMQConnection.cpp#L1329].] These exceptions become ActiveMQ exceptions [here|[https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/ActiveMQConnection.cpp#L1257],] and then to CMS Exceptions [here|[https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/kernels/ActiveMQConsumerKernel.cpp#L1426].] The only exceptions caught in [ActiveMQSessionKernel::acknowledge|https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/kernels/ActiveMQSessionKernel.cpp#L508] are [decaf exceptions|https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/kernels/ActiveMQSessionKernel.cpp#L518]; when a CMS exception is caught, the [consumer read lock|https://github.com/apache/activemq-cpp/blob/master/activemq-cpp/src/main/activemq/core/kernels/ActiveMQSessionKernel.cpp#L510] is not released.

This issue can be fixed by catching the CMS exceptions.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)