You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by dampudia <da...@tecsisa.com> on 2009/08/31 18:00:32 UTC
Camel threads deadlock
Hello:
I am developing a process using camel routes (v. 1.4.0) inside servicemix,
and, with some frequency, I found that all camel threads are waiting and my
process is locked.
The process flow is, more or less:
- First I have a big message, with a list of elements that must be
processed.
- I use a splitter to divide the message and send the parts to another
endpoint (the beginning of another router).
- That router makes some operations, invoking some JBI services.
- Finally it creates a response-per-part and sends them to an ActiveMQ
queue.
I have 16 threads and, using JConsole, I see that 15 of them are locked with
trace [1]
and the last one has trace [2]
I know the question is far more generic it should, but I am completely lost
and dont know what to do. It seems that some internal queue is full, that
most of my threads are sleeping while trying to insert messages in that
queue and that nobody is emptying the queue.
I would like to know if someone has ever seen a similar behaviour, and if
there is an usual suspect to watch closely.
Thank you very much in advance.
dampudia
-----------------------------------------------------------------------------------------------
[1] sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:1841)
java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:368)
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound
(DeliveryChannelImpl.java:673)
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask
(ThreadPoolExecutor.java:650)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
java.lang.Thread.run(Thread.java:595)
[2] sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
(AbstractQueuedSynchronizer.java:716)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptib
ly(AbstractQueuedSynchronizer.java:872)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly
(AbstractQueuedSynchronizer.java:1179)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:199)
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:49)
org.apache.camel.processor.DelegateAsyncProcessor.process
(DelegateAsyncProcessor.java:66)
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:47)
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:61)
org.apache.camel.processor.MulticastProcessor.process
(MulticastProcessor.java:185)
org.apache.camel.management.InstrumentationProcessor.process
(InstrumentationProcessor.java:69)
org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:155)
org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:91)
org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
org.apache.camel.processor.Pipeline.process(Pipeline.java:85)
[...]
--
View this message in context: http://www.nabble.com/Camel-threads-deadlock-tp25225906p25225906.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel threads deadlock
Posted by dampudia <da...@tecsisa.com>.
Finally it seems that the cause of this problem was the use of
ServicemixClient: the so generated exchanges couldnt be returned to their
creator and got parked in the queue.
The solution has been the transformation of the creator class to a
<bean:endpoint>, implementing MessageExchangeListener and using a
deliveryChannel to create the exchanges and send them. Then, when the
exchanges end their processing, they return to the bean, where I set "done"
status, finishing correctly.
dampudia wrote:
>
> Luckily, I´ve been able to reduce greatly the scope of the problem (and it
> has little relation to what I previouly said).
>
> The problem is, mainly, that I´m using a ServicemixClient object (obtained
> using a ClientFactory, obtained through JNDI) to create InOnly exchanges I
> send to the beginning of a camel pipeline. But these exchanges doesnt have
> the behaviour I expected: they end the process correctly, but they get
> "done" status, get stuck in the camel inbound queue (an
> ArrayBlockingQueue) and dont get out of there. When 1024 messages are
> inside the queue, the threads cannot insert anymore and get locked,
> paralyzing the entire component.
>
> Now I´m trying to avoid that, and any idea would be welcome.
>
> Anyway, thank you very much for your time and effort.
>
>
> Stan Lewis-3 wrote:
>>
>> On Mon, Aug 31, 2009 at 12:34 PM, dampudia<da...@tecsisa.com> wrote:
>>>
>>> The version is 3.3, and the route works fine in my test suite (where all
>>> endpoints apart from pipeline´s are mocked). The SEDA queue max size is
>>> 256,
>>> but the problem is not here (when the error appears it is not full, just
>>> the
>>> threads are locked and following messages get enqueued).
>>>
>>> And I cannot change from SedaFlow to JMSFlow because of the app
>>> environment.
>>
>> Gotcha. Does the behavior change at all if you bump up the number of
>> threads in servicemix? What other servicemix components are you using
>> as part of this flow?
>>
>>>
>>> Stan Lewis-3 wrote:
>>>>
>>>> What version of servicemix are you running this in? Does the route
>>>> work okay outside of servicemix (maybe replace the calls to your JBI
>>>> services with some mock endpoint)? Have you checked how many elements
>>>> are in the SEDA queue that the NMR is using, or have you perhaps tried
>>>> switching from SedaFlow to JMSFlow to see if that helps things?
>>>>
>>>> On Mon, Aug 31, 2009 at 12:00 PM, dampudia<da...@tecsisa.com> wrote:
>>>>>
>>>>> Hello:
>>>>>
>>>>> I am developing a process using camel routes (v. 1.4.0) inside
>>>>> servicemix,
>>>>> and, with some frequency, I found that all camel threads are waiting
>>>>> and
>>>>> my
>>>>> process is locked.
>>>>>
>>>>> The process flow is, more or less:
>>>>>
>>>>> - First I have a big message, with a list of elements that must be
>>>>> processed.
>>>>> - I use a splitter to divide the message and send the parts to another
>>>>> endpoint (the beginning of another router).
>>>>> - That router makes some operations, invoking some JBI services.
>>>>> - Finally it creates a response-per-part and sends them to an ActiveMQ
>>>>> queue.
>>>>>
>>>>> I have 16 threads and, using JConsole, I see that 15 of them are
>>>>> locked
>>>>> with
>>>>> trace [1]
>>>>> and the last one has trace [2]
>>>>>
>>>>> I know the question is far more generic it should, but I am completely
>>>>> lost
>>>>> and dont know what to do. It seems that some internal queue is full,
>>>>> that
>>>>> most of my threads are sleeping while trying to insert messages in
>>>>> that
>>>>> queue and that nobody is emptying the queue.
>>>>>
>>>>> I would like to know if someone has ever seen a similar behaviour, and
>>>>> if
>>>>> there is an usual suspect to watch closely.
>>>>>
>>>>> Thank you very much in advance.
>>>>>
>>>>> dampudia
>>>>>
>>>>> -----------------------------------------------------------------------------------------------
>>>>>
>>>>> [1] sun.misc.Unsafe.park(Native Method)
>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
>>>>> (AbstractQueuedSynchronizer.java:1841)
>>>>> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:368)
>>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound
>>>>> (DeliveryChannelImpl.java:673)
>>>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
>>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask
>>>>> (ThreadPoolExecutor.java:650)
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>>>>> java.lang.Thread.run(Thread.java:595)
>>>>>
>>>>> [2] sun.misc.Unsafe.park(Native Method)
>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
>>>>> (AbstractQueuedSynchronizer.java:716)
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptib
>>>>> ly(AbstractQueuedSynchronizer.java:872)
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly
>>>>> (AbstractQueuedSynchronizer.java:1179)
>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:199)
>>>>> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:49)
>>>>> org.apache.camel.processor.DelegateAsyncProcessor.process
>>>>> (DelegateAsyncProcessor.java:66)
>>>>> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:47)
>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:61)
>>>>> org.apache.camel.processor.MulticastProcessor.process
>>>>> (MulticastProcessor.java:185)
>>>>> org.apache.camel.management.InstrumentationProcessor.process
>>>>> (InstrumentationProcessor.java:69)
>>>>> org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:155)
>>>>> org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:91)
>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:85)
>>>>> [...]
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Camel-threads-deadlock-tp25225906p25225906.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Camel-threads-deadlock-tp25225906p25226418.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Camel-threads-deadlock-tp25225906p25243961.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel threads deadlock
Posted by dampudia <da...@tecsisa.com>.
Luckily, I´ve been able to reduce greatly the scope of the problem (and it
has little relation to what I previouly said).
The problem is, mainly, that I´m using a ServicemixClient object (obtained
using a ClientFactory, obtained through JNDI) to create InOnly exchanges I
send to the beginning of a camel pipeline. But these exchanges doesnt have
the behaviour I expected: they end the process correctly, but they get
"done" status, get stuck in the camel inbound queue (an ArrayBlockingQueue)
and dont get out of there. When 1024 messages are inside the queue, the
threads cannot insert anymore and get locked, paralyzing the entire
component.
Now I´m trying to avoid that, and any idea would be welcome.
Anyway, thank you very much for your time and effort.
Stan Lewis-3 wrote:
>
> On Mon, Aug 31, 2009 at 12:34 PM, dampudia<da...@tecsisa.com> wrote:
>>
>> The version is 3.3, and the route works fine in my test suite (where all
>> endpoints apart from pipeline´s are mocked). The SEDA queue max size is
>> 256,
>> but the problem is not here (when the error appears it is not full, just
>> the
>> threads are locked and following messages get enqueued).
>>
>> And I cannot change from SedaFlow to JMSFlow because of the app
>> environment.
>
> Gotcha. Does the behavior change at all if you bump up the number of
> threads in servicemix? What other servicemix components are you using
> as part of this flow?
>
>>
>> Stan Lewis-3 wrote:
>>>
>>> What version of servicemix are you running this in? Does the route
>>> work okay outside of servicemix (maybe replace the calls to your JBI
>>> services with some mock endpoint)? Have you checked how many elements
>>> are in the SEDA queue that the NMR is using, or have you perhaps tried
>>> switching from SedaFlow to JMSFlow to see if that helps things?
>>>
>>> On Mon, Aug 31, 2009 at 12:00 PM, dampudia<da...@tecsisa.com> wrote:
>>>>
>>>> Hello:
>>>>
>>>> I am developing a process using camel routes (v. 1.4.0) inside
>>>> servicemix,
>>>> and, with some frequency, I found that all camel threads are waiting
>>>> and
>>>> my
>>>> process is locked.
>>>>
>>>> The process flow is, more or less:
>>>>
>>>> - First I have a big message, with a list of elements that must be
>>>> processed.
>>>> - I use a splitter to divide the message and send the parts to another
>>>> endpoint (the beginning of another router).
>>>> - That router makes some operations, invoking some JBI services.
>>>> - Finally it creates a response-per-part and sends them to an ActiveMQ
>>>> queue.
>>>>
>>>> I have 16 threads and, using JConsole, I see that 15 of them are locked
>>>> with
>>>> trace [1]
>>>> and the last one has trace [2]
>>>>
>>>> I know the question is far more generic it should, but I am completely
>>>> lost
>>>> and dont know what to do. It seems that some internal queue is full,
>>>> that
>>>> most of my threads are sleeping while trying to insert messages in that
>>>> queue and that nobody is emptying the queue.
>>>>
>>>> I would like to know if someone has ever seen a similar behaviour, and
>>>> if
>>>> there is an usual suspect to watch closely.
>>>>
>>>> Thank you very much in advance.
>>>>
>>>> dampudia
>>>>
>>>> -----------------------------------------------------------------------------------------------
>>>>
>>>> [1] sun.misc.Unsafe.park(Native Method)
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
>>>> (AbstractQueuedSynchronizer.java:1841)
>>>> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:368)
>>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound
>>>> (DeliveryChannelImpl.java:673)
>>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
>>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask
>>>> (ThreadPoolExecutor.java:650)
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>>>> java.lang.Thread.run(Thread.java:595)
>>>>
>>>> [2] sun.misc.Unsafe.park(Native Method)
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
>>>> (AbstractQueuedSynchronizer.java:716)
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptib
>>>> ly(AbstractQueuedSynchronizer.java:872)
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly
>>>> (AbstractQueuedSynchronizer.java:1179)
>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:199)
>>>> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:49)
>>>> org.apache.camel.processor.DelegateAsyncProcessor.process
>>>> (DelegateAsyncProcessor.java:66)
>>>> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:47)
>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:61)
>>>> org.apache.camel.processor.MulticastProcessor.process
>>>> (MulticastProcessor.java:185)
>>>> org.apache.camel.management.InstrumentationProcessor.process
>>>> (InstrumentationProcessor.java:69)
>>>> org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:155)
>>>> org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:91)
>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:85)
>>>> [...]
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Camel-threads-deadlock-tp25225906p25225906.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Camel-threads-deadlock-tp25225906p25226418.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Camel-threads-deadlock-tp25225906p25241781.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel threads deadlock
Posted by Stan Lewis <ga...@gmail.com>.
On Mon, Aug 31, 2009 at 12:34 PM, dampudia<da...@tecsisa.com> wrote:
>
> The version is 3.3, and the route works fine in my test suite (where all
> endpoints apart from pipeline´s are mocked). The SEDA queue max size is 256,
> but the problem is not here (when the error appears it is not full, just the
> threads are locked and following messages get enqueued).
>
> And I cannot change from SedaFlow to JMSFlow because of the app environment.
Gotcha. Does the behavior change at all if you bump up the number of
threads in servicemix? What other servicemix components are you using
as part of this flow?
>
> Stan Lewis-3 wrote:
>>
>> What version of servicemix are you running this in? Does the route
>> work okay outside of servicemix (maybe replace the calls to your JBI
>> services with some mock endpoint)? Have you checked how many elements
>> are in the SEDA queue that the NMR is using, or have you perhaps tried
>> switching from SedaFlow to JMSFlow to see if that helps things?
>>
>> On Mon, Aug 31, 2009 at 12:00 PM, dampudia<da...@tecsisa.com> wrote:
>>>
>>> Hello:
>>>
>>> I am developing a process using camel routes (v. 1.4.0) inside
>>> servicemix,
>>> and, with some frequency, I found that all camel threads are waiting and
>>> my
>>> process is locked.
>>>
>>> The process flow is, more or less:
>>>
>>> - First I have a big message, with a list of elements that must be
>>> processed.
>>> - I use a splitter to divide the message and send the parts to another
>>> endpoint (the beginning of another router).
>>> - That router makes some operations, invoking some JBI services.
>>> - Finally it creates a response-per-part and sends them to an ActiveMQ
>>> queue.
>>>
>>> I have 16 threads and, using JConsole, I see that 15 of them are locked
>>> with
>>> trace [1]
>>> and the last one has trace [2]
>>>
>>> I know the question is far more generic it should, but I am completely
>>> lost
>>> and dont know what to do. It seems that some internal queue is full, that
>>> most of my threads are sleeping while trying to insert messages in that
>>> queue and that nobody is emptying the queue.
>>>
>>> I would like to know if someone has ever seen a similar behaviour, and if
>>> there is an usual suspect to watch closely.
>>>
>>> Thank you very much in advance.
>>>
>>> dampudia
>>>
>>> -----------------------------------------------------------------------------------------------
>>>
>>> [1] sun.misc.Unsafe.park(Native Method)
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
>>> (AbstractQueuedSynchronizer.java:1841)
>>> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:368)
>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound
>>> (DeliveryChannelImpl.java:673)
>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask
>>> (ThreadPoolExecutor.java:650)
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>>> java.lang.Thread.run(Thread.java:595)
>>>
>>> [2] sun.misc.Unsafe.park(Native Method)
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
>>> (AbstractQueuedSynchronizer.java:716)
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptib
>>> ly(AbstractQueuedSynchronizer.java:872)
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly
>>> (AbstractQueuedSynchronizer.java:1179)
>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:199)
>>> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:49)
>>> org.apache.camel.processor.DelegateAsyncProcessor.process
>>> (DelegateAsyncProcessor.java:66)
>>> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:47)
>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:61)
>>> org.apache.camel.processor.MulticastProcessor.process
>>> (MulticastProcessor.java:185)
>>> org.apache.camel.management.InstrumentationProcessor.process
>>> (InstrumentationProcessor.java:69)
>>> org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:155)
>>> org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:91)
>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:85)
>>> [...]
>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Camel-threads-deadlock-tp25225906p25225906.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Camel-threads-deadlock-tp25225906p25226418.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
Re: Camel threads deadlock
Posted by dampudia <da...@tecsisa.com>.
The version is 3.3, and the route works fine in my test suite (where all
endpoints apart from pipeline´s are mocked). The SEDA queue max size is 256,
but the problem is not here (when the error appears it is not full, just the
threads are locked and following messages get enqueued).
And I cannot change from SedaFlow to JMSFlow because of the app environment.
Stan Lewis-3 wrote:
>
> What version of servicemix are you running this in? Does the route
> work okay outside of servicemix (maybe replace the calls to your JBI
> services with some mock endpoint)? Have you checked how many elements
> are in the SEDA queue that the NMR is using, or have you perhaps tried
> switching from SedaFlow to JMSFlow to see if that helps things?
>
> On Mon, Aug 31, 2009 at 12:00 PM, dampudia<da...@tecsisa.com> wrote:
>>
>> Hello:
>>
>> I am developing a process using camel routes (v. 1.4.0) inside
>> servicemix,
>> and, with some frequency, I found that all camel threads are waiting and
>> my
>> process is locked.
>>
>> The process flow is, more or less:
>>
>> - First I have a big message, with a list of elements that must be
>> processed.
>> - I use a splitter to divide the message and send the parts to another
>> endpoint (the beginning of another router).
>> - That router makes some operations, invoking some JBI services.
>> - Finally it creates a response-per-part and sends them to an ActiveMQ
>> queue.
>>
>> I have 16 threads and, using JConsole, I see that 15 of them are locked
>> with
>> trace [1]
>> and the last one has trace [2]
>>
>> I know the question is far more generic it should, but I am completely
>> lost
>> and dont know what to do. It seems that some internal queue is full, that
>> most of my threads are sleeping while trying to insert messages in that
>> queue and that nobody is emptying the queue.
>>
>> I would like to know if someone has ever seen a similar behaviour, and if
>> there is an usual suspect to watch closely.
>>
>> Thank you very much in advance.
>>
>> dampudia
>>
>> -----------------------------------------------------------------------------------------------
>>
>> [1] sun.misc.Unsafe.park(Native Method)
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
>> (AbstractQueuedSynchronizer.java:1841)
>> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:368)
>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound
>> (DeliveryChannelImpl.java:673)
>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask
>> (ThreadPoolExecutor.java:650)
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>> java.lang.Thread.run(Thread.java:595)
>>
>> [2] sun.misc.Unsafe.park(Native Method)
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
>> (AbstractQueuedSynchronizer.java:716)
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptib
>> ly(AbstractQueuedSynchronizer.java:872)
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly
>> (AbstractQueuedSynchronizer.java:1179)
>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:199)
>> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:49)
>> org.apache.camel.processor.DelegateAsyncProcessor.process
>> (DelegateAsyncProcessor.java:66)
>> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:47)
>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:61)
>> org.apache.camel.processor.MulticastProcessor.process
>> (MulticastProcessor.java:185)
>> org.apache.camel.management.InstrumentationProcessor.process
>> (InstrumentationProcessor.java:69)
>> org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:155)
>> org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:91)
>> org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
>> org.apache.camel.processor.Pipeline.process(Pipeline.java:85)
>> [...]
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Camel-threads-deadlock-tp25225906p25225906.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Camel-threads-deadlock-tp25225906p25226418.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel threads deadlock
Posted by Stan Lewis <ga...@gmail.com>.
What version of servicemix are you running this in? Does the route
work okay outside of servicemix (maybe replace the calls to your JBI
services with some mock endpoint)? Have you checked how many elements
are in the SEDA queue that the NMR is using, or have you perhaps tried
switching from SedaFlow to JMSFlow to see if that helps things?
On Mon, Aug 31, 2009 at 12:00 PM, dampudia<da...@tecsisa.com> wrote:
>
> Hello:
>
> I am developing a process using camel routes (v. 1.4.0) inside servicemix,
> and, with some frequency, I found that all camel threads are waiting and my
> process is locked.
>
> The process flow is, more or less:
>
> - First I have a big message, with a list of elements that must be
> processed.
> - I use a splitter to divide the message and send the parts to another
> endpoint (the beginning of another router).
> - That router makes some operations, invoking some JBI services.
> - Finally it creates a response-per-part and sends them to an ActiveMQ
> queue.
>
> I have 16 threads and, using JConsole, I see that 15 of them are locked with
> trace [1]
> and the last one has trace [2]
>
> I know the question is far more generic it should, but I am completely lost
> and dont know what to do. It seems that some internal queue is full, that
> most of my threads are sleeping while trying to insert messages in that
> queue and that nobody is emptying the queue.
>
> I would like to know if someone has ever seen a similar behaviour, and if
> there is an usual suspect to watch closely.
>
> Thank you very much in advance.
>
> dampudia
>
> -----------------------------------------------------------------------------------------------
>
> [1] sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:1841)
> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:368)
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound
> (DeliveryChannelImpl.java:673)
> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> (ThreadPoolExecutor.java:650)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> java.lang.Thread.run(Thread.java:595)
>
> [2] sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
> (AbstractQueuedSynchronizer.java:716)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptib
> ly(AbstractQueuedSynchronizer.java:872)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly
> (AbstractQueuedSynchronizer.java:1179)
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:199)
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:49)
> org.apache.camel.processor.DelegateAsyncProcessor.process
> (DelegateAsyncProcessor.java:66)
> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:47)
> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:61)
> org.apache.camel.processor.MulticastProcessor.process
> (MulticastProcessor.java:185)
> org.apache.camel.management.InstrumentationProcessor.process
> (InstrumentationProcessor.java:69)
> org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:155)
> org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:91)
> org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
> org.apache.camel.processor.Pipeline.process(Pipeline.java:85)
> [...]
>
>
> --
> View this message in context: http://www.nabble.com/Camel-threads-deadlock-tp25225906p25225906.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>