Camel splitter threadpool configuration : messages stuck in queue ?

[A577127 <an...@worldline.com>]

Hi everyone,

I'm working with camel to build a server that involves a lot of parallel
processing, and some of my messages get stuck somewhere. I will try to
explain my problem by explaining what I'm doing.

I have this camel route :



The idea is that the REST request that triggers the route contains an array
of tokens. The splitter component creates 1 message/token and gives it to
APNS. For example, I send 1 request with 10000 tokens. My APNS component
should receive 10000 camel messages and send 10000 notifications.

It works fine, however the splitter creates each message when the previous
one has finished the route. I saw that the  camel-splitter
<http://camel.apache.org/splitter.html>   had settings to allow parallel
processing. At first I just set  in my route. It still works fine, but I
wasn't happy with the default settings of the thread pool. That's where I
got stuck.


First, I tryed to set the MaxQueueSize of the defaultThreadPoolProfile of my
camelContext :



Setting the maxQueueSize to -1 causes some messages to get stuck. For
example, When I send my 10000 tokens, the APNS component receives ~2000
messages, and the 8000 others are stuck somewhere.

Then I tryed to set the threadpool of the splitter manually in my route
configuration, this way :



This brings the same issue, some messages are lost somewhere.


Any ideas to solve this problem ? Thanks.



--
View this message in context: http://camel.465427.n5.nabble.com/Camel-splitter-threadpool-configuration-messages-stuck-in-queue-tp5754194.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel splitter threadpool configuration : messages stuck in queue ?

[A577127 <an...@worldline.com>]

Thanks for your reply.

Uh ok, I put the wrong log4j level and didn't saw all the
OutOfMemoryErrors...

Using Executors.newFixedThreadPool(1000) looks good. However as you pointed
out, I'm looking for the best number of threads. Am I forced to define a
static (max) size ?

I hoped that the cachedThreadPool could create threads until it runs out of
memory, and queue the next ones for later. Its java doc sounds great for my
case :




--
View this message in context: http://camel.465427.n5.nabble.com/Camel-splitter-threadpool-configuration-messages-stuck-in-queue-tp5754194p5754216.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel splitter threadpool configuration : messages stuck in queue ?

[Ravi Godbole <ra...@cognizant.com>]

Are you getting any memory specific issues which indicate inability of system
to create indefinite threads ?

It seems you want to spawn as many threads as possible in order to execute
the job faster.

Have you tried following option and played around with number of threads in
the pool ? [ in the following case it is set to 1000 )

.split().method(MySplitterBean.class,"split").executorService(Executors.newFixedThreadPool(1000))





--
View this message in context: http://camel.465427.n5.nabble.com/Camel-splitter-threadpool-configuration-messages-stuck-in-queue-tp5754194p5754207.html
Sent from the Camel - Users mailing list archive at Nabble.com.