You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Alexander Ilyin <al...@madadsmedia.com> on 2015/04/20 15:19:39 UTC
RecipientList thread pool problem
Hi,
I have a problem with RecipientList pattern. I'm using it in a parallel
processing mode which means it is backed up by a thread pool. The
configuration is as follows:
<threadPool id="pool" threadName="My RecipientList" poolSize="10"
maxPoolSize="200" />
<recipientList id="bidderResponseGatherer"
strategyRef="gatherBidderResponses" parallelProcessing="true"
stopOnException="false" timeout="1000" ignoreInvalidEndpoints="true"
executorServiceRef="pool" >
<header>RecipientsList</header>
</recipientList>
The problem is that pool threads aren't released properly sometimes. After
a few hours following the start of an application I can see using jstack
tool that some threads are stuck in a WAITING state:
"Camel (mainContext) thread #187456 - NettyOrderedWorker" daemon prio=10
tid=0x00007fb14c058000 nid=0x5564 waiting on condition [0x00007fb2472d0000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007fb4dc319940> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at
org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:327)
at
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:213)
at
org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:153)
at
org.apache.camel.processor.RecipientList.process(RecipientList.java:112)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at
org.apache.camel.component.netty.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:136)
at
org.apache.camel.component.netty.handlers.ServerChannelHandler.messageReceived(ServerChannelHandler.java:108)
at
org.apache.camel.component.netty.http.handlers.HttpServerChannelHandler.messageReceived(HttpServerChannelHandler.java:199)
at
org.apache.camel.component.netty.http.handlers.HttpServerMultiplexChannelHandler.messageReceived(HttpServerMultiplexChannelHandler.java:103)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
at
org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
at
org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Number of such threads grows gradually until the application isn't able to
respond to the requests. I've found very similar ticket in your Jira (
https://issues.apache.org/jira/browse/CAMEL-6717) but it is stated there
that the problem is in the 3rd party library mqtt-client. However, I'm not
using this library, I'm only using Netty to make http requests but still
see a very similar behaviour.
I've updated Apache Camel to the latest version 2.15.1 (from 2.12.2) as
suggested here
http://camel.465427.n5.nabble.com/Camel-Netty-ReadTimeout-event-not-trigerred-td5761676.html.
But this didn't help too. Also I've noticed that thread is becoming
unavailable after the timeout specified for the calls has been reached.
Although not every timeout leads to a thread leak.
Can you suggest what else can be checked? Am I missing something? Or may be
this is a bug in Camel?
--
Alexander Ilyin
*tel: 856-874-8928 <856-874-8928>* Director of Engineering
alexander.i@madadsmedia.com 717 Fellowship Rd. Suite D
Mount Laurel, NJ 08054
Re: RecipientList thread pool problem
Posted by Alexander Ilyin <al...@madadsmedia.com>.
Hi Claus,
thank you for your answer.
I hadn't configured netty request timeout before I wrote my first letter.
Now I've read that tasks may keep running when using RecipientList timeout
and have set up netty request timeout instead of RecipientList timeout as
you suggested. But the situation didn't change much. NettyOrderedWorker
threads are still getting stuck in a WAITING state after some time
following the start of an app as well as RecipientList threads.
What else can be checked? I can provide additional info if needed.
On Mon, Apr 20, 2015 at 4:19 PM, Alexander Ilyin <
alexander.i@madadsmedia.com> wrote:
> Hi,
>
> I have a problem with RecipientList pattern. I'm using it in a parallel
> processing mode which means it is backed up by a thread pool. The
> configuration is as follows:
>
> <threadPool id="pool" threadName="My RecipientList" poolSize="10"
> maxPoolSize="200" />
>
> <recipientList id="bidderResponseGatherer"
> strategyRef="gatherBidderResponses" parallelProcessing="true"
> stopOnException="false" timeout="1000" ignoreInvalidEndpoints="true"
> executorServiceRef="pool" >
> <header>RecipientsList</header>
> </recipientList>
>
> The problem is that pool threads aren't released properly sometimes. After
> a few hours following the start of an application I can see using jstack
> tool that some threads are stuck in a WAITING state:
>
> "Camel (mainContext) thread #187456 - NettyOrderedWorker" daemon prio=10
> tid=0x00007fb14c058000 nid=0x5564 waiting on condition [0x00007fb2472d0000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007fb4dc319940> (a
> java.util.concurrent.CountDownLatch$Sync)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:327)
> at
> org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:213)
> at
> org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:153)
> at
> org.apache.camel.processor.RecipientList.process(RecipientList.java:112)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
> at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at
> org.apache.camel.component.netty.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:136)
> at
> org.apache.camel.component.netty.handlers.ServerChannelHandler.messageReceived(ServerChannelHandler.java:108)
> at
> org.apache.camel.component.netty.http.handlers.HttpServerChannelHandler.messageReceived(HttpServerChannelHandler.java:199)
> at
> org.apache.camel.component.netty.http.handlers.HttpServerMultiplexChannelHandler.messageReceived(HttpServerMultiplexChannelHandler.java:103)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
> at
> org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Number of such threads grows gradually until the application isn't able to
> respond to the requests. I've found very similar ticket in your Jira (
> https://issues.apache.org/jira/browse/CAMEL-6717) but it is stated there
> that the problem is in the 3rd party library mqtt-client. However, I'm not
> using this library, I'm only using Netty to make http requests but still
> see a very similar behaviour.
>
> I've updated Apache Camel to the latest version 2.15.1 (from 2.12.2) as
> suggested here
> http://camel.465427.n5.nabble.com/Camel-Netty-ReadTimeout-event-not-trigerred-td5761676.html.
> But this didn't help too. Also I've noticed that thread is becoming
> unavailable after the timeout specified for the calls has been reached.
> Although not every timeout leads to a thread leak.
>
> Can you suggest what else can be checked? Am I missing something? Or may
> be this is a bug in Camel?
>
>
> --
> Alexander Ilyin
> *tel: 856-874-8928 <856-874-8928>* Director of Engineering
> alexander.i@madadsmedia.com 717 Fellowship Rd. Suite D
> Mount Laurel, NJ 08054
>
--
Alexander Ilyin
*tel: 856-874-8928 <856-874-8928>* Director of Engineering
alexander.i@madadsmedia.com 717 Fellowship Rd. Suite D
Mount Laurel, NJ 08054
Re: RecipientList thread pool problem
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Have you configured netty to use request timeout?
http://camel.apache.org/netty.html
You should favor using that for timeout, and not as much the timeout
on the recipient list. Or make sure those values are so netty timeout
first.
On Mon, Apr 20, 2015 at 3:19 PM, Alexander Ilyin
<al...@madadsmedia.com> wrote:
> Hi,
>
> I have a problem with RecipientList pattern. I'm using it in a parallel
> processing mode which means it is backed up by a thread pool. The
> configuration is as follows:
>
> <threadPool id="pool" threadName="My RecipientList" poolSize="10"
> maxPoolSize="200" />
>
> <recipientList id="bidderResponseGatherer"
> strategyRef="gatherBidderResponses" parallelProcessing="true"
> stopOnException="false" timeout="1000" ignoreInvalidEndpoints="true"
> executorServiceRef="pool" >
> <header>RecipientsList</header>
> </recipientList>
>
> The problem is that pool threads aren't released properly sometimes. After
> a few hours following the start of an application I can see using jstack
> tool that some threads are stuck in a WAITING state:
>
> "Camel (mainContext) thread #187456 - NettyOrderedWorker" daemon prio=10
> tid=0x00007fb14c058000 nid=0x5564 waiting on condition [0x00007fb2472d0000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007fb4dc319940> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:327)
> at
> org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:213)
> at
> org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:153)
> at
> org.apache.camel.processor.RecipientList.process(RecipientList.java:112)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
> at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at
> org.apache.camel.component.netty.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:136)
> at
> org.apache.camel.component.netty.handlers.ServerChannelHandler.messageReceived(ServerChannelHandler.java:108)
> at
> org.apache.camel.component.netty.http.handlers.HttpServerChannelHandler.messageReceived(HttpServerChannelHandler.java:199)
> at
> org.apache.camel.component.netty.http.handlers.HttpServerMultiplexChannelHandler.messageReceived(HttpServerMultiplexChannelHandler.java:103)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
> at
> org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Number of such threads grows gradually until the application isn't able to
> respond to the requests. I've found very similar ticket in your Jira (
> https://issues.apache.org/jira/browse/CAMEL-6717) but it is stated there
> that the problem is in the 3rd party library mqtt-client. However, I'm not
> using this library, I'm only using Netty to make http requests but still
> see a very similar behaviour.
>
> I've updated Apache Camel to the latest version 2.15.1 (from 2.12.2) as
> suggested here
> http://camel.465427.n5.nabble.com/Camel-Netty-ReadTimeout-event-not-trigerred-td5761676.html.
> But this didn't help too. Also I've noticed that thread is becoming
> unavailable after the timeout specified for the calls has been reached.
> Although not every timeout leads to a thread leak.
>
> Can you suggest what else can be checked? Am I missing something? Or may be
> this is a bug in Camel?
>
>
> --
> Alexander Ilyin
> *tel: 856-874-8928 <856-874-8928>* Director of Engineering
> alexander.i@madadsmedia.com 717 Fellowship Rd. Suite D
> Mount Laurel, NJ 08054
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/