[jira] Commented: (SM-585) Deadlock on BoundedLinkedQueue

["Guillaume Nodet (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/activemq/browse/SM-585?page=comments#action_36993 ] 
            
Guillaume Nodet commented on SM-585:
------------------------------------

It should work much better now with SM-594.

Instead of using a single WorkManager, I have refactored things to use multiple thread pools based on java.util.concurrent
work.  These thread pools are configurable, and should degrade nicely.
I need to do more load testing, but it seems far better now.

I'll let you close this issue ...

> Deadlock on BoundedLinkedQueue
> ------------------------------
>
>                 Key: SM-585
>                 URL: https://issues.apache.org/activemq/browse/SM-585
>             Project: ServiceMix
>          Issue Type: Bug
>          Components: servicemix-jbi
>    Affects Versions: 3.0
>         Environment: Windows XP, Java 5
>            Reporter: Robin Byrne
>         Assigned To: Guillaume Nodet
>            Priority: Critical
>         Attachments: deadlock.zip
>
>
> We've encountered a problem with slow endpoints leading to a deadlock situation around the BoundedLinkedQueue. We've reproduce the problem with Seda, JMS and JCA flows.
> The situation is similar to that described for SM-512/SM-521 in the Jira, although we don't think sendSync is necessarily involved.
> Our servicemix configuration started with a JMS endpoint receiving messages from a topic.  The messages were sent to a pipeline which invoked an XSL transform and forwarded the result to an HTTPComponent. The HTTPComponent sent the transformed messages to a SOAP service.
> We noticed that rapidly sending a large number of messages to the JMS topic would lock up ServiceMix. That is,  
> 1) no more messages were sent to the SOAP service
> 2) more messages would simply create more blocked threads
> 3) ServiceMix never recovered.
> We suspected that the slowness of the HTTP interaction was triggering the problem.  To confirm this we replaced the HTTPComponent with a TraceComponent - modified to wait a configurable number of millis before sending the done message.
> We were able to reliably recreate the problem.  We found that, in the blocked state, all of the worker threads were waiting to put ME's into the BoundedLinkedQueue.  Many of the threads were blocked trying to put 'done' messages into the queue.
> We found that, as long as the Queue for the TraceComponent remained below capacity, that system continued to work. However, when the queue for that component hit capacity (100 MEs) the Trace output stopped.  Even if the inbound flow of messages stopped at that point, and all of the other queue's were empty, the queue for the TraceComponent retained those 100 MEs
> After that, the next 100 inbound messages simply filled up the prior Queue (for the XSLT) until it hit capacity.  And so it went until all of Queues were full. Once all of the Queue's were full each new message created a thread that blocked trying to put the new ME into the first Queue.
> The attached zip file contains the servicemix config file, xslt, and modified TraceComponent.java necessary to recreate the problem. You will also need a test harness capable of pushing a lot of messages into the JMS topic.  A burst of 250 messages onto the JMS topic should do it.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira