Deadlock condition in BatchProcessor$BatchSender

[jboyens <jb...@fooninja.org>]

Hi.

I'm really hoping I'm wrong, but I was doing some load testing with Camel
and an aggregator coming off a asyncSendBody to a direct: component. 

I've set the ExecutorThreadPool on the producer so that I can watch what's
going on inside and eventually at about 16 msgs / second the threads start
to lock on the queue's Reentrant lock and deadlock. I've checked with jstack
and confirmed that they lock up on that queue lock.

It looks as though the run() method holds the lock while the enqueueExchange
method attempts to get the same lock and fails because the run() method
never lets it go.

There is a hole that opens up when sendExchanges() runs and the run() method
gives up the lock. So, exchanges will get enqueued at low speeds, but at
higher speeds the bulk up.
-- 
View this message in context: http://old.nabble.com/Deadlock-condition-in-BatchProcessor%24BatchSender-tp26822335p26822335.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Deadlock condition in BatchProcessor$BatchSender

[Martin Krasser <de...@martin-krasser.de>]

Hi,

sorry for the late answer. Can you please provide the stacktraces for 
the two threads that deadlock? Best you create a JIRA issue with that 
information plus a sample application or unit test that represents the 
setup/application you're using (even if the deadlock is hard to 
reproduce or not reproducible at all in a unit test).

jboyens schrieb:
> Hi.
>
> I'm really hoping I'm wrong, but I was doing some load testing with Camel
> and an aggregator coming off a asyncSendBody to a direct: component. 
>
> I've set the ExecutorThreadPool on the producer so that I can watch what's
> going on inside and eventually at about 16 msgs / second the threads start
> to lock on the queue's Reentrant lock and deadlock. I've checked with jstack
> and confirmed that they lock up on that queue lock.
>
> It looks as though the run() method holds the lock while the enqueueExchange
> method attempts to get the same lock and fails because the run() method
> never lets it go.
>
> There is a hole that opens up when sendExchanges() runs and the run() method
> gives up the lock. So, exchanges will get enqueued at low speeds, but at
> higher speeds the bulk up.
>   

Re: Deadlock condition in BatchProcessor$BatchSender

[Claus Ibsen <cl...@gmail.com>]

On Thu, Dec 17, 2009 at 3:51 AM, jboyens <jb...@fooninja.org> wrote:
>
> Hi.
>
> I'm really hoping I'm wrong, but I was doing some load testing with Camel
> and an aggregator coming off a asyncSendBody to a direct: component.
>
> I've set the ExecutorThreadPool on the producer so that I can watch what's
> going on inside and eventually at about 16 msgs / second the threads start
> to lock on the queue's Reentrant lock and deadlock. I've checked with jstack
> and confirmed that they lock up on that queue lock.
>
> It looks as though the run() method holds the lock while the enqueueExchange
> method attempts to get the same lock and fails because the run() method
> never lets it go.
>
> There is a hole that opens up when sendExchanges() runs and the run() method
> gives up the lock. So, exchanges will get enqueued at low speeds, but at
> higher speeds the bulk up.
> --

As Martin said we would love some more details from you if possible.

That said, we got a fair amount of tickets to rework the Aggregator
EIP pattern which is build on top of BatchProcessor.
And in that ballpark we may rework BatchProcessor as well. I am not
found of the low level code that tries to be smart and only hold locks
when really needed. Often you are not smarter than the JVM and its
easy to miss something and very hard to test that it just works under
all conditions. And you may have to account for different OS behave
differently. E.g. does it also run smoothly withy IBM JDK on AIX etc?

But I do not think we got the time to do this (Aggregator) within the
2.2 timeframe so a patch / fix for this issue is much welcome.



> View this message in context: http://old.nabble.com/Deadlock-condition-in-BatchProcessor%24BatchSender-tp26822335p26822335.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus