You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by damianharvey <da...@gmail.com> on 2010/11/23 03:01:42 UTC
Errors when under load
I'm doing some load testing and finding that I get the occasional error.
These are intermittent and as far as I can tell they are unrelated to my
code. To me they look like config / tuning issues. Can anyone tell me what I
might need to configure?
The errors are:
java.util.ConcurrentModificationException
at java.util.AbstractList$ListItr.set(Unknown Source)[:1.6.0_13]
at java.util.Collections.sort(Unknown Source)[:1.6.0_13]
at
org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:42)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:172)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.processor.DefaultChannel$1.done(DefaultChannel.java:262)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:330)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:208)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:256)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:169)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:111)[camel-core-2.5.0.jar:2.5.0]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)[:1.6.0_13]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)[:1.6.0_13]
at java.lang.Thread.run(Unknown Source)[:1.6.0_13]
And
javax.management.InstanceAlreadyExistsException:
org.apache.camel:context=aon43098/camelContext,type=threadpools,name=ThreadPoolExecutor(0xf848fa)
at com.sun.jmx.mbeanserver.Repository.addMBean(Unknown Source)[:1.6.0_13]
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(Unknown
Source)[:1.6.0_13]
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(Unknown
Source)[:1.6.0_13]
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(Unknown
Source)[:1.6.0_13]
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(Unknown
Source)[:1.6.0_13]
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(Unknown
Source)[:1.6.0_13]
at
org.apache.camel.management.DefaultManagementAgent.registerMBeanWithServer(DefaultManagementAgent.java:320)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.management.DefaultManagementAgent.register(DefaultManagementAgent.java:228)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.management.DefaultManagementAgent.register(DefaultManagementAgent.java:211)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.management.ManagedManagementStrategy.manageNamedObject(ManagedManagementStrategy.java:75)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.management.ManagedManagementStrategy.manageObject(ManagedManagementStrategy.java:68)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.management.DefaultManagementLifecycleStrategy.onThreadPoolAdd(DefaultManagementLifecycleStrategy.java:559)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.impl.DefaultExecutorServiceStrategy.onThreadPoolCreated(DefaultExecutorServiceStrategy.java:334)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.impl.DefaultExecutorServiceStrategy.newThreadPool(DefaultExecutorServiceStrategy.java:281)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.impl.DefaultExecutorServiceStrategy.newDefaultThreadPool(DefaultExecutorServiceStrategy.java:185)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.impl.DefaultProducerTemplate.doStart(DefaultProducerTemplate.java:737)[camel-core-2.5.0.jar:2.5.0]
at
org.apache.camel.impl.ServiceSupport.start(ServiceSupport.java:65)[camel-core-2.5.0.jar:2.5.0]
The latter message goes away if I use a ThreadPoolProfile such as :
<threadPoolProfile id="aonCbsDefaultThreadProfile" defaultProfile="true"
poolSize="30" maxPoolSize="50" maxQueueSize="1000"
rejectedPolicy="CallerRuns" />
In addition if I open VisualVM and look at the Heap I can see that Camel is
not garbage collecting and it eventually blows the heap. Is there some Camel
resource that I should be careful to close/flush? Anything around the
Producer/Consumer templates?
Thanks,
Damian.
--
View this message in context: http://camel.465427.n5.nabble.com/Errors-when-under-load-tp3276259p3276259.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Errors when under load
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I have committed a fix on trunk. You are welcome to test the fix on your system.
On Tue, Nov 23, 2010 at 10:15 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> Thanks for reporting. I have created a ticket
> https://issues.apache.org/activemq/browse/CAMEL-3355
>
>
> On Tue, Nov 23, 2010 at 3:01 AM, damianharvey
> <da...@gmail.com> wrote:
>>
>> I'm doing some load testing and finding that I get the occasional error.
>> These are intermittent and as far as I can tell they are unrelated to my
>> code. To me they look like config / tuning issues. Can anyone tell me what I
>> might need to configure?
>>
>> The errors are:
>>
>> java.util.ConcurrentModificationException
>> at java.util.AbstractList$ListItr.set(Unknown Source)[:1.6.0_13]
>> at java.util.Collections.sort(Unknown Source)[:1.6.0_13]
>> at
>> org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:42)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:172)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.processor.DefaultChannel$1.done(DefaultChannel.java:262)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:330)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:208)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:256)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:169)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:111)[camel-core-2.5.0.jar:2.5.0]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
>> Source)[:1.6.0_13]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>> Source)[:1.6.0_13]
>> at java.lang.Thread.run(Unknown Source)[:1.6.0_13]
>>
>>
>> And
>>
>>
>> javax.management.InstanceAlreadyExistsException:
>> org.apache.camel:context=aon43098/camelContext,type=threadpools,name=ThreadPoolExecutor(0xf848fa)
>> at com.sun.jmx.mbeanserver.Repository.addMBean(Unknown Source)[:1.6.0_13]
>> at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(Unknown
>> Source)[:1.6.0_13]
>> at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(Unknown
>> Source)[:1.6.0_13]
>> at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(Unknown
>> Source)[:1.6.0_13]
>> at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(Unknown
>> Source)[:1.6.0_13]
>> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(Unknown
>> Source)[:1.6.0_13]
>> at
>> org.apache.camel.management.DefaultManagementAgent.registerMBeanWithServer(DefaultManagementAgent.java:320)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.management.DefaultManagementAgent.register(DefaultManagementAgent.java:228)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.management.DefaultManagementAgent.register(DefaultManagementAgent.java:211)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.management.ManagedManagementStrategy.manageNamedObject(ManagedManagementStrategy.java:75)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.management.ManagedManagementStrategy.manageObject(ManagedManagementStrategy.java:68)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.management.DefaultManagementLifecycleStrategy.onThreadPoolAdd(DefaultManagementLifecycleStrategy.java:559)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.impl.DefaultExecutorServiceStrategy.onThreadPoolCreated(DefaultExecutorServiceStrategy.java:334)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.impl.DefaultExecutorServiceStrategy.newThreadPool(DefaultExecutorServiceStrategy.java:281)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.impl.DefaultExecutorServiceStrategy.newDefaultThreadPool(DefaultExecutorServiceStrategy.java:185)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.impl.DefaultProducerTemplate.doStart(DefaultProducerTemplate.java:737)[camel-core-2.5.0.jar:2.5.0]
>> at
>> org.apache.camel.impl.ServiceSupport.start(ServiceSupport.java:65)[camel-core-2.5.0.jar:2.5.0]
>>
>>
>> The latter message goes away if I use a ThreadPoolProfile such as :
>> <threadPoolProfile id="aonCbsDefaultThreadProfile" defaultProfile="true"
>> poolSize="30" maxPoolSize="50" maxQueueSize="1000"
>> rejectedPolicy="CallerRuns" />
>>
>>
>> In addition if I open VisualVM and look at the Heap I can see that Camel is
>> not garbage collecting and it eventually blows the heap. Is there some Camel
>> resource that I should be careful to close/flush? Anything around the
>> Producer/Consumer templates?
>>
>> Thanks,
>>
>> Damian.
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/Errors-when-under-load-tp3276259p3276259.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: Errors when under load
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Thanks for reporting. I have created a ticket
https://issues.apache.org/activemq/browse/CAMEL-3355
On Tue, Nov 23, 2010 at 3:01 AM, damianharvey
<da...@gmail.com> wrote:
>
> I'm doing some load testing and finding that I get the occasional error.
> These are intermittent and as far as I can tell they are unrelated to my
> code. To me they look like config / tuning issues. Can anyone tell me what I
> might need to configure?
>
> The errors are:
>
> java.util.ConcurrentModificationException
> at java.util.AbstractList$ListItr.set(Unknown Source)[:1.6.0_13]
> at java.util.Collections.sort(Unknown Source)[:1.6.0_13]
> at
> org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:42)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:172)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.processor.UnitOfWorkProcessor.doneUow(UnitOfWorkProcessor.java:121)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.processor.UnitOfWorkProcessor.access$000(UnitOfWorkProcessor.java:36)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.processor.UnitOfWorkProcessor$1.done(UnitOfWorkProcessor.java:106)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.processor.DefaultChannel$1.done(DefaultChannel.java:262)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:330)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:208)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:256)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:169)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:111)[camel-core-2.5.0.jar:2.5.0]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)[:1.6.0_13]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)[:1.6.0_13]
> at java.lang.Thread.run(Unknown Source)[:1.6.0_13]
>
>
> And
>
>
> javax.management.InstanceAlreadyExistsException:
> org.apache.camel:context=aon43098/camelContext,type=threadpools,name=ThreadPoolExecutor(0xf848fa)
> at com.sun.jmx.mbeanserver.Repository.addMBean(Unknown Source)[:1.6.0_13]
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(Unknown
> Source)[:1.6.0_13]
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(Unknown
> Source)[:1.6.0_13]
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(Unknown
> Source)[:1.6.0_13]
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(Unknown
> Source)[:1.6.0_13]
> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(Unknown
> Source)[:1.6.0_13]
> at
> org.apache.camel.management.DefaultManagementAgent.registerMBeanWithServer(DefaultManagementAgent.java:320)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.management.DefaultManagementAgent.register(DefaultManagementAgent.java:228)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.management.DefaultManagementAgent.register(DefaultManagementAgent.java:211)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.management.ManagedManagementStrategy.manageNamedObject(ManagedManagementStrategy.java:75)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.management.ManagedManagementStrategy.manageObject(ManagedManagementStrategy.java:68)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.management.DefaultManagementLifecycleStrategy.onThreadPoolAdd(DefaultManagementLifecycleStrategy.java:559)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.impl.DefaultExecutorServiceStrategy.onThreadPoolCreated(DefaultExecutorServiceStrategy.java:334)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.impl.DefaultExecutorServiceStrategy.newThreadPool(DefaultExecutorServiceStrategy.java:281)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.impl.DefaultExecutorServiceStrategy.newDefaultThreadPool(DefaultExecutorServiceStrategy.java:185)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.impl.DefaultProducerTemplate.doStart(DefaultProducerTemplate.java:737)[camel-core-2.5.0.jar:2.5.0]
> at
> org.apache.camel.impl.ServiceSupport.start(ServiceSupport.java:65)[camel-core-2.5.0.jar:2.5.0]
>
>
> The latter message goes away if I use a ThreadPoolProfile such as :
> <threadPoolProfile id="aonCbsDefaultThreadProfile" defaultProfile="true"
> poolSize="30" maxPoolSize="50" maxQueueSize="1000"
> rejectedPolicy="CallerRuns" />
>
>
> In addition if I open VisualVM and look at the Heap I can see that Camel is
> not garbage collecting and it eventually blows the heap. Is there some Camel
> resource that I should be careful to close/flush? Anything around the
> Producer/Consumer templates?
>
> Thanks,
>
> Damian.
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Errors-when-under-load-tp3276259p3276259.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/