You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@camel.apache.org by "Vanshul.Chawla" <Va...@target.com> on 2015/04/07 09:58:13 UTC

Issue with high volume load in camel

Hello,

I have a camel context where the messages are picked from a IBM MQ queue and on basis of XPATH, we place it onto separate seda queues.
>From seda queues, the messages are picked and files are written in per minute duration ie append mode for 1 minute.

This camel context has peak volumes of 10 million messages per day.

When we start the context, it runs fine for 5 10 minutes and then slows down and finally stops with error:

org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[JmsMessage@0x3f197cf8]
                at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1379)
                at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:283)
                at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:64)
                at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
                at org.apache.camel.processor.interceptor.HandleFaultInterceptor.process(HandleFaultInterceptor.java:41)
                at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:91)
                at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
                at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
                at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
                at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
                at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
                at org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:291)
                at org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:200)
                at org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:147)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Could you please suggest some pointers here. We don't have any java code except for a small class which removes tab,linefeed from message before dumping that to file.


Thanks and Regards,

Vanshul Chawla

RE: Issue with high volume load in camel

Posted by "Vanshul.Chawla" <Va...@target.com>.

Yes. It was able to take load of 150K with one context. Started 2 in parallel to test higher loads.

Thanks and Regards,

Vanshul Chawla


-----Original Message-----
From: yogu13 [mailto:yogu13@gmail.com] 
Sent: Wednesday, April 08, 2015 11:29 AM
To: users@camel.apache.org
Subject: RE: Issue with high volume load in camel

"Started 2 contexts together and it worked.  "

Does this mean your issue is resolved?




--
View this message in context: http://camel.465427.n5.nabble.com/Issue-with-high-volume-load-in-camel-tp5765430p5765465.html
Sent from the Camel - Users mailing list archive at Nabble.com.

RE: Issue with high volume load in camel

Posted by yogu13 <yo...@gmail.com>.

"Started 2 contexts together and it worked.  "

Does this mean your issue is resolved?




--
View this message in context: http://camel.465427.n5.nabble.com/Issue-with-high-volume-load-in-camel-tp5765430p5765465.html
Sent from the Camel - Users mailing list archive at Nabble.com.

RE: Issue with high volume load in camel

Posted by "Vanshul.Chawla" <Va...@target.com>.

Thanks. We tested with queue to file and queue to seda and then file for 150K message per hour. It works fine. Beyond that, it starts slowing in 5 10 min in both cases.
Started 2 contexts together and it worked.

Thanks and Regards,

Vanshul Chawla

-----Original Message-----
From: yogu13 [mailto:yogu13@gmail.com] 
Sent: Wednesday, April 08, 2015 11:05 AM
To: users@camel.apache.org
Subject: Re: Issue with high volume load in camel

Incase you need to seda for any reason then setting queueSize and blockWhenFull properties on  Seda <http://camel.apache.org/seda.html>
could also help!




Regards,
-Yogesh



--
View this message in context: http://camel.465427.n5.nabble.com/Issue-with-high-volume-load-in-camel-tp5765430p5765462.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Issue with high volume load in camel

Posted by yogu13 <yo...@gmail.com>.

Incase you need to seda for any reason then setting queueSize and
blockWhenFull properties on  Seda <http://camel.apache.org/seda.html>  
could also help!




Regards,
-Yogesh



--
View this message in context: http://camel.465427.n5.nabble.com/Issue-with-high-volume-load-in-camel-tp5765430p5765462.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Issue with high volume load in camel

Posted by yogu13 <yo...@gmail.com>.

I wouldnt choose to pick the messages from the queue and write it to seda and
instead write it to files directly after picking up from the queue. Seda
seems to be an overhead in this case.

Just my thought.

Regards,
-Yogesh







--
View this message in context: http://camel.465427.n5.nabble.com/Issue-with-high-volume-load-in-camel-tp5765430p5765461.html
Sent from the Camel - Users mailing list archive at Nabble.com.