You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by dcheckoway <dc...@gmail.com> on 2011/04/08 13:32:10 UTC
ConcurrentModificationException in embedded 5.5.0 broker
Hello,
We just migrated from ActiveMQ 5.4.2 to ActiveMQ 5.5.0. So far so good,
with one exception (pun not intended). In one case where we have an
embedded broker, we're seeing this exception get logged on occasion:
WARNING; 08-Apr-2011 11:11:41; tid:45931; TransportConnection stopAsync;
cannot create async transport stopper thread.. not waiting for stop to
complete, reason:
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
at java.util.HashMap$EntryIterator.next(HashMap.java:834)
at java.util.HashMap$EntryIterator.next(HashMap.java:832)
at java.util.HashMap.putAllForCreate(HashMap.java:435)
at java.util.HashMap.(HashMap.java:225)
at
org.slf4j.helpers.BasicMDCAdapter.getCopyOfContextMap(BasicMDCAdapter.java:130)
at org.slf4j.MDC.getCopyOfContextMap(MDC.java:182)
at
org.apache.activemq.util.MDCHelper.getCopyOfContextMap(MDCHelper.java:30)
at
org.apache.activemq.broker.TransportConnection.stopAsync(TransportConnection.java:946)
at
org.apache.activemq.broker.TransportConnection.processShutdown(TransportConnection.java:353)
at
org.apache.activemq.command.ShutdownInfo.visit(ShutdownInfo.java:35)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:218)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:127)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Up until this morning, I had only seen that logged during shutdown of the
app context. But just now, it popped out independently, out of the blue.
For what it's worth, here's the app context config:
" producerFlowControl="false" memoryLimit="20mb"/>
Has anybody else seen this ConcurrentModificationException happening with
5.5.0 (or otherwise)? Any ideas?
-----
Dan Checkoway
dcheckoway gmail com
--
View this message in context: http://activemq.2283324.n4.nabble.com/ConcurrentModificationException-in-embedded-5-5-0-broker-tp3435952p3435952.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ConcurrentModificationException in embedded 5.5.0 broker
Posted by Geoffrey Arnold <ge...@gmail.com>.
We're having the same issue with SLF4J 1.5.11 (the version distributed with ActiveMQ 5.5.0).
On Apr 8, 2011, at 7:35 AM, dcheckoway <dc...@gmail.com> wrote:
> I suppose I should also mention that we're using slf4j 1.6.1. Not sure if
> that has anything to do with this, since the stack trace does show it
> happening in slf4j land...
>
>
> org.slf4j
> slf4j-api
> 1.6.1
> runtime
>
>
> org.slf4j
> slf4j-jdk14
> 1.6.1
> runtime
>
>
> -----
> Dan Checkoway
> dcheckoway gmail com
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ConcurrentModificationException-in-embedded-5-5-0-broker-tp3435952p3435958.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ConcurrentModificationException in embedded 5.5.0 broker
Posted by dcheckoway <dc...@gmail.com>.
I suppose I should also mention that we're using slf4j 1.6.1. Not sure if
that has anything to do with this, since the stack trace does show it
happening in slf4j land...
org.slf4j
slf4j-api
1.6.1
runtime
org.slf4j
slf4j-jdk14
1.6.1
runtime
-----
Dan Checkoway
dcheckoway gmail com
--
View this message in context: http://activemq.2283324.n4.nabble.com/ConcurrentModificationException-in-embedded-5-5-0-broker-tp3435952p3435958.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ConcurrentModificationException in embedded 5.5.0 broker
Posted by dcheckoway <dc...@gmail.com>.
Hey, just curious what the strategy for the next release is. I can't use
5.5.0 in production because this ConcurrentModificationException bug (full
stack trace below) is causing message loss.
Will there be a 5.5.1 very soon, pretty please?! When 5.5.0 comes out and
there is a known issue that causes message loss...and that issue gets fixed
VERY quickly (thanks Gary!)...but you don't issue a bug fix release shortly
thereafter...something is wrong with this picture.
We're literally still running 5.4.1 in production because 5.4.2 has its
kahadb journal file missing/corrupt issues (fixed in 5.5.0), but we can't
use 5.5.0 because of this recent bug that causes message loss.
Please consider putting out a point release, because 5.5.0 seems to be top
notch other than this known and already fixed (fix version 5.6.0?!?!) issue.
Or are there other major issues with 5.5.0 that cropped up and haven't been
fixed yet?
Just trying to understand the release strategy and set expectations
internally at my company. (And the word "kestrel" keeps coming up.)
Thanks,
Dan
org.springframework.jms.UncategorizedJmsException: Uncategorized exception
occured during JMS processing; nested exception is javax.jms.JMSException:
Could not create Transport. Reason:
java.util.ConcurrentModificationException
at
org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
at
org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
at
org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)
at
org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:170)
at
org.apache.camel.component.jms.JmsProducer.doSend(JmsProducer.java:355)
at
org.apache.camel.component.jms.JmsProducer.processInOnly(JmsProducer.java:309)
at
org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:99)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77)
at
org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:104)
at
org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:272)
at
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:98)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:99)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:299)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:208)
at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:269)
at
org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:109)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:103)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:85)
at
org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:92)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.jms.JMSException: Could not create Transport. Reason:
java.util.ConcurrentModificationException
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
at
org.apache.activemq.ActiveMQConnectionFactory.createTransport(ActiveMQConnectionFactory.java:245)
at
org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:258)
at
org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:230)
at
org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:178)
at
org.apache.activemq.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:137)
at
org.apache.activemq.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:121)
at
org.apache.activemq.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:92)
at
org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
at
org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:456)
... 39 more
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
at java.util.HashMap$EntryIterator.next(HashMap.java:834)
at java.util.HashMap$EntryIterator.next(HashMap.java:832)
at java.util.HashMap.putAllForCreate(HashMap.java:435)
at java.util.HashMap.<init>(HashMap.java:225)
at
org.slf4j.helpers.BasicMDCAdapter.getCopyOfContextMap(BasicMDCAdapter.java:130)
at org.slf4j.MDC.getCopyOfContextMap(MDC.java:182)
at
org.apache.activemq.util.MDCHelper.getCopyOfContextMap(MDCHelper.java:30)
at
org.apache.activemq.thread.PooledTaskRunner.<init>(PooledTaskRunner.java:42)
at
org.apache.activemq.thread.TaskRunnerFactory.createTaskRunner(TaskRunnerFactory.java:80)
at
org.apache.activemq.transport.failover.FailoverTransport.<init>(FailoverTransport.java:129)
at
org.apache.activemq.transport.failover.FailoverTransportFactory.createTransport(FailoverTransportFactory.java:71)
at
org.apache.activemq.transport.failover.FailoverTransportFactory.createTransport(FailoverTransportFactory.java:62)
at
org.apache.activemq.transport.failover.FailoverTransportFactory.doConnect(FailoverTransportFactory.java:37)
at
org.apache.activemq.transport.TransportFactory.doConnect(TransportFactory.java:51)
at
org.apache.activemq.transport.TransportFactory.connect(TransportFactory.java:80)
at
org.apache.activemq.ActiveMQConnectionFactory.createTransport(ActiveMQConnectionFactory.java:243)
... 47 more
-----
Dan Checkoway
dcheckoway gmail com
--
View this message in context: http://activemq.2283324.n4.nabble.com/ConcurrentModificationException-in-embedded-5-5-0-broker-tp3435952p3467903.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ConcurrentModificationException in embedded 5.5.0 broker
Posted by dcheckoway <dc...@gmail.com>.
@Gary, you bet: https://issues.apache.org/jira/browse/AMQ-3276
-----
Dan Checkoway
dcheckoway gmail com
--
View this message in context: http://activemq.2283324.n4.nabble.com/ConcurrentModificationException-in-embedded-5-5-0-broker-tp3435952p3436324.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ConcurrentModificationException in embedded 5.5.0 broker
Posted by Gary Tully <ga...@gmail.com>.
I think there is a problem with the way the context is referenced in
amq when threads are created, it should do nothing in that case
really. investigating now.
Can you create a jira issue to track this?
On 8 April 2011 12:32, dcheckoway <dc...@gmail.com> wrote:
> Hello,
>
> We just migrated from ActiveMQ 5.4.2 to ActiveMQ 5.5.0. So far so good,
> with one exception (pun not intended). In one case where we have an
> embedded broker, we're seeing this exception get logged on occasion:
>
> WARNING; 08-Apr-2011 11:11:41; tid:45931; TransportConnection stopAsync;
> cannot create async transport stopper thread.. not waiting for stop to
> complete, reason:
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$EntryIterator.next(HashMap.java:834)
> at java.util.HashMap$EntryIterator.next(HashMap.java:832)
> at java.util.HashMap.putAllForCreate(HashMap.java:435)
> at java.util.HashMap.(HashMap.java:225)
> at
> org.slf4j.helpers.BasicMDCAdapter.getCopyOfContextMap(BasicMDCAdapter.java:130)
> at org.slf4j.MDC.getCopyOfContextMap(MDC.java:182)
> at
> org.apache.activemq.util.MDCHelper.getCopyOfContextMap(MDCHelper.java:30)
> at
> org.apache.activemq.broker.TransportConnection.stopAsync(TransportConnection.java:946)
> at
> org.apache.activemq.broker.TransportConnection.processShutdown(TransportConnection.java:353)
> at
> org.apache.activemq.command.ShutdownInfo.visit(ShutdownInfo.java:35)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
> at
> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:218)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:127)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Up until this morning, I had only seen that logged during shutdown of the
> app context. But just now, it popped out independently, out of the blue.
>
> For what it's worth, here's the app context config:
>
>
>
>
>
>
> " producerFlowControl="false" memoryLimit="20mb"/>
>
>
>
>
>
>
>
>
> Has anybody else seen this ConcurrentModificationException happening with
> 5.5.0 (or otherwise)? Any ideas?
>
> -----
> Dan Checkoway
> dcheckoway gmail com
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ConcurrentModificationException-in-embedded-5-5-0-broker-tp3435952p3435952.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://blog.garytully.com
http://fusesource.com