You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Jayson Minard (JIRA)" <ji...@apache.org> on 2008/10/08 20:42:52 UTC
[jira] Created: (AMQ-1971) ConcurrentModificationException in high
volume broker
ConcurrentModificationException in high volume broker
------------------------------------------------------
Key: AMQ-1971
URL: https://issues.apache.org/activemq/browse/AMQ-1971
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.2.0
Reporter: Jayson Minard
This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AMQ-1971)
ConcurrentModificationException in high volume broker
Posted by "Jayson Minard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46388#action_46388 ]
jayson.minard edited comment on AMQ-1971 at 10/12/08 11:58 PM:
---------------------------------------------------------------
I have a tested patch for the ConcurrentModificationException in AMQ-1971, this also uncovered a bug with a related class. See below:
Patch to fix ConcurrentModificationException which is caused by not having a perfectly synchronized this.iterator and this.batchList in AbstractStoreCursor. Also removed any access to these member variables from subclasses to ensure they are synchronized and make sure they are not manipulated seperately. This caused a small change in TopicStorePrefetch which had a reference to batchList although it actually overrode the ancestor class which meant that its remove() method never actually worked. So that was replaced with a call to AbstractSoreCursor to do the actual remove work. A few TODO comments are left in the patch for review by someone that knows more.
Basically, the fix is to ensure that all changes to batchList cause a release of the cached iterator.
I apologize in advance for reformatting the code before generating the patch. I did that midway through the edit and it is a constant habit so it caused some noise in the patch. If you apply it, then diff while ignoring white-space most of the noise is removed.
was (Author: jayson.minard):
I have a tested patch for the ConcurrentModificationException in AMQ-1971, this also uncovered a bug with a related class. See below:
Patch to fix ConcurrentModificationException which is caused by not having a perfectly synchronized this.iterator and this.batchList in AbstractStoreCursor. Also removed any access to these member variables from subclasses to ensure they are synchronized and make sure they are not manipulated seperately. This caused a small change in TopicStorePrefetch which had a reference to batchList although it actually overrode the ancestor class which meant that its remove() method never actually worked. So that was replaced with a call to AbstractSoreCursor to do the actual remove work. A few TODO comments are left in the patch for review by someone that knows more.
Basically, the fix is to ensure that all changes to batchList cause a release of the cached iterator.
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
> Assignee: Rob Davies
> Attachments: amq1971.patch
>
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (AMQ-1971) ConcurrentModificationException in high
volume broker
Posted by "Rob Davies (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rob Davies reassigned AMQ-1971:
-------------------------------
Assignee: Rob Davies
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
> Assignee: Rob Davies
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1971) ConcurrentModificationException in
high volume broker
Posted by "Jayson Minard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46374#action_46374 ]
Jayson Minard commented on AMQ-1971:
------------------------------------
This is on 5.2.0 built from SVN tag of the same name. Client producer/consumers use POM 5.2.0 from staging server.
Bug also exists when running on 1 broker without a network of brokers.
The series of queues basically are via Camel:
File -> parsing -> queue
queue -> normalization -> queue
queue -> database transactions -> queue
queue -> external systems
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1971) ConcurrentModificationException in
high volume broker
Posted by "Jayson Minard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46393#action_46393 ]
Jayson Minard commented on AMQ-1971:
------------------------------------
Hey Rob, thanks for making a fix. A quick question about the TopicStorePrefetch descendant class. It instantiates its own batchList which then hides the ancestor one. IT looks like you removed the dispatched() method which eliminates the previous issue I fixed and makes that part irrelevant. But should the private batchList still be there in TopicStorePrefetch since it now appears to be unused?
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
> Assignee: Rob Davies
> Fix For: 5.3.0
>
> Attachments: amq1971.patch
>
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1971) ConcurrentModificationException in
high volume broker
Posted by "Jayson Minard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46384#action_46384 ]
Jayson Minard commented on AMQ-1971:
------------------------------------
Queues form admin console look like this at time of failure (note that the admin UI now truncates queue names for display making most unreadable and unusable in the display). Note that some queue sizes are negative numbers which we've seen before but not tracked down to anything specific.
<queues>
<queue name="mag.query1.queue.record.physical">
<stats size="-2" consumerCount="10" enqueueCount="29610" dequeueCount="29612"/>
<feed>
<atom>queueBrowse/mag.query1.queue.record.physical?view=rss&feedType=atom_1.0</atom>
<rss>queueBrowse/mag.query1.queue.record.physical?view=rss&feedType=rss_2.0</rss>
</feed>
</queue>
<queue name="mag.query1.queue.record.physical.pdf">
<stats size="6" consumerCount="0" enqueueCount="6" dequeueCount="0"/>
<feed>
<atom>queueBrowse/mag.query1.queue.record.physical.pdf?view=rss&feedType=atom_1.0</atom>
<rss>queueBrowse/mag.query1.queue.record.physical.pdf?view=rss&feedType=rss_2.0</rss>
</feed>
</queue>
<queue name="mag.queue.record.search.zh">
<stats size="0" consumerCount="10" enqueueCount="10" dequeueCount="10"/>
<feed>
<atom>queueBrowse/mag.queue.record.search.zh?view=rss&feedType=atom_1.0</atom>
<rss>queueBrowse/mag.queue.record.search.zh?view=rss&feedType=rss_2.0</rss>
</feed>
</queue>
<queue name="mag.queue.record.search.en">
<stats size="-7" consumerCount="10" enqueueCount="29602" dequeueCount="29609"/>
<feed>
<atom>queueBrowse/mag.queue.record.search.en?view=rss&feedType=atom_1.0</atom>
<rss>queueBrowse/mag.queue.record.search.en?view=rss&feedType=rss_2.0</rss>
</feed>
</queue>
<queue name="mag.Events.Errors">
<stats size="0" consumerCount="1" enqueueCount="8" dequeueCount="8"/>
<feed>
<atom>queueBrowse/mag.Events.Errors?view=rss&feedType=atom_1.0</atom>
<rss>queueBrowse/mag.Events.Errors?view=rss&feedType=rss_2.0</rss>
</feed>
</queue>
<queue name="mag.queue.record.logical">
<stats size="0" consumerCount="1" enqueueCount="29612" dequeueCount="29612"/>
<feed>
<atom>queueBrowse/mag.queue.record.logical?view=rss&feedType=atom_1.0</atom>
<rss>queueBrowse/mag.queue.record.logical?view=rss&feedType=rss_2.0</rss>
</feed>
</queue>
<queue name="test.existingRecordQueue">
<stats size="0" consumerCount="1" enqueueCount="0" dequeueCount="0"/>
<feed>
<atom>queueBrowse/test.existingRecordQueue?view=rss&feedType=atom_1.0</atom>
<rss>queueBrowse/test.existingRecordQueue?view=rss&feedType=rss_2.0</rss>
</feed>
</queue>
<queue name="mag.query1.queue.record.raw">
<stats size="138407" consumerCount="50" enqueueCount="168029" dequeueCount="29622"/>
<feed>
<atom>queueBrowse/mag.query1.queue.record.raw?view=rss&feedType=atom_1.0</atom>
<rss>queueBrowse/mag.query1.queue.record.raw?view=rss&feedType=rss_2.0</rss>
</feed>
</queue>
<queue name="example.A">
<stats size="0" consumerCount="1" enqueueCount="0" dequeueCount="0"/>
<feed>
<atom>queueBrowse/example.A?view=rss&feedType=atom_1.0</atom>
<rss>queueBrowse/example.A?view=rss&feedType=rss_2.0</rss>
</feed>
</queue>
<queue name="mag.query1.queue.record.physical.augment">
<stats size="0" consumerCount="1" enqueueCount="27311" dequeueCount="27311"/>
<feed>
<atom>queueBrowse/mag.query1.queue.record.physical.augment?view=rss&feedType=atom_1.0</atom>
<rss>queueBrowse/mag.query1.queue.record.physical.augment?view=rss&feedType=rss_2.0</rss>
</feed>
</queue>
<queue name="mag.queue.record.search.other">
<stats size="0" consumerCount="10" enqueueCount="0" dequeueCount="0"/>
<feed>
<atom>queueBrowse/mag.queue.record.search.other?view=rss&feedType=atom_1.0</atom>
<rss>queueBrowse/mag.queue.record.search.other?view=rss&feedType=rss_2.0</rss>
</feed>
</queue>
</queues>
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-1971) ConcurrentModificationException in high
volume broker
Posted by "Jayson Minard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jayson Minard updated AMQ-1971:
-------------------------------
Attachment: amq1971.patch
I have a tested patch for the ConcurrentModificationException in AMQ-1971, this also uncovered a bug with a related class. See below:
Patch to fix ConcurrentModificationException which is caused by not having a perfectly synchronized this.iterator and this.batchList in AbstractStoreCursor. Also removed any access to these member variables from subclasses to ensure they are synchronized and make sure they are not manipulated seperately. This caused a small change in TopicStorePrefetch which had a reference to batchList although it actually overrode the ancestor class which meant that its remove() method never actually worked. So that was replaced with a call to AbstractSoreCursor to do the actual remove work. A few TODO comments are left in the patch for review by someone that knows more.
Basically, the fix is to ensure that all changes to batchList cause a release of the cached iterator.
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
> Assignee: Rob Davies
> Attachments: amq1971.patch
>
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AMQ-1971)
ConcurrentModificationException in high volume broker
Posted by "Jayson Minard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46373#action_46373 ]
jayson.minard edited comment on AMQ-1971 at 10/11/08 3:37 PM:
--------------------------------------------------------------
More recent flavor of the same:
ERROR Queue - Failed to page in more queue messages
java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
INFO HashIndex - Resize hash bins data_queue#3a#2f#2fmag.query1.queue.record.physical.augment from 1024 to 2048
was (Author: jayson.minard):
More recent flavor of the same:
ERROR Queue - Failed to page in more queue messages
java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
INFO HashIndex - Resize hash bins data_queue#3a#2f#2fmagnolia.query1.queue.record.physical.augment from 1024 to 2048
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1971) ConcurrentModificationException in
high volume broker
Posted by "Jayson Minard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46382#action_46382 ]
Jayson Minard commented on AMQ-1971:
------------------------------------
Trying it now. I'll pull his binaries and make sure the client updates (should already via the POM)
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AMQ-1971)
ConcurrentModificationException in high volume broker
Posted by "Jayson Minard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46374#action_46374 ]
jayson.minard edited comment on AMQ-1971 at 10/11/08 3:57 PM:
--------------------------------------------------------------
This is on 5.2.0 built from SVN tag of the same name. Client producer/consumers use POM 5.2.0 from staging server. JDK 1.6 on Linux.
Bug also exists when running on 1 broker without a network of brokers.
The series of queues basically are via Camel:
File -> parsing -> queue
queue -> normalization -> queue
queue -> database transactions -> queue
queue -> external systems
was (Author: jayson.minard):
This is on 5.2.0 built from SVN tag of the same name. Client producer/consumers use POM 5.2.0 from staging server.
Bug also exists when running on 1 broker without a network of brokers.
The series of queues basically are via Camel:
File -> parsing -> queue
queue -> normalization -> queue
queue -> database transactions -> queue
queue -> external systems
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AMQ-1971) ConcurrentModificationException in high
volume broker
Posted by "Rob Davies (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rob Davies resolved AMQ-1971.
-----------------------------
Fix Version/s: 5.3.0
Resolution: Fixed
Fixed by SVN revision 703975
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
> Assignee: Rob Davies
> Fix For: 5.3.0
>
> Attachments: amq1971.patch
>
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1971) ConcurrentModificationException in
high volume broker
Posted by "Jayson Minard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46373#action_46373 ]
Jayson Minard commented on AMQ-1971:
------------------------------------
More recent flavor of the same:
ERROR Queue - Failed to page in more queue messages
java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
INFO HashIndex - Resize hash bins data_queue#3a#2f#2fmagnolia.query1.queue.record.physical.augment from 1024 to 2048
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1971) ConcurrentModificationException in
high volume broker
Posted by "Hans Bausewein (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46376#action_46376 ]
Hans Bausewein commented on AMQ-1971:
-------------------------------------
Have you tried RC2?
See http://www.nabble.com/-VOTE--ActiveMQ-5.2.0---RC2-td19874910.html
If it fails, vote -1 and add the link to this issue.
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AMQ-1971)
ConcurrentModificationException in high volume broker
Posted by "Jayson Minard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46373#action_46373 ]
jayson.minard edited comment on AMQ-1971 at 10/11/08 3:37 PM:
--------------------------------------------------------------
More recent flavor of the same, but with a hash bin resize message nearby in case that is relevant.
ERROR Queue - Failed to page in more queue messages
java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
INFO HashIndex - Resize hash bins data_queue#3a#2f#2fmag.query1.queue.record.physical.augment from 1024 to 2048
was (Author: jayson.minard):
More recent flavor of the same:
ERROR Queue - Failed to page in more queue messages
java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
INFO HashIndex - Resize hash bins data_queue#3a#2f#2fmag.query1.queue.record.physical.augment from 1024 to 2048
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1971) ConcurrentModificationException in
high volume broker
Posted by "Jayson Minard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46383#action_46383 ]
Jayson Minard commented on AMQ-1971:
------------------------------------
Still have error on single broker run... JDK 1.6, ACTIVEMQ_OPTS=-Xmx1g, pretty vanilla activemq.xml configuration other than changing broker name and adding a few settings here to the broker element for testing:
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="${HOSTNAME}" dataDirectory="${activemq.base}/data"
keepDurableSubsActive="true" useJmx="true" useVirtualTopics="false" deleteAllMessagesOnStartup="true" >
INFO log - jetty-6.1.9
INFO WebConsoleStarter - ActiveMQ WebConsole initialized.
INFO /admin - Initializing Spring FrameworkServlet 'dispatcher'
INFO log - ActiveMQ Console at http://0.0.0.0:8161/admin
INFO log - ActiveMQ Web Demos at http://0.0.0.0:8161/demo
INFO log - RESTful file access application at http://0.0.0.0:8161/fileserver
INFO log - Started SelectChannelConnector@0.0.0.0:8161
WARN BrokerRegistry - Broker localhost not started so using query1.removed.removed.com instead
INFO TransportConnector - Connector vm://localhost Started
ERROR Queue - Failed to page in more queue messages
java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1181)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ERROR Queue - Failed to page in more queue messages
java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1181)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ERROR Queue - Failed to page in more queue messages
java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1181)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ERROR Queue - Failed to page in more queue messages
java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1181)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
> ConcurrentModificationException in high volume broker
> ------------------------------------------------------
>
> Key: AMQ-1971
> URL: https://issues.apache.org/activemq/browse/AMQ-1971
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Jayson Minard
>
> This is occurring in the thousands as messages come through. We have a network of 5 brokers although they do most of their work internally (publisher/consumer for queue are only on same broker) and do less work crossing to a central broker.
> java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.next(AbstractStoreCursor.java:136)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.next(StoreQueueCursor.java:140)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1178)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1290)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1004)
> at org.apache.activemq.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
> at org.apache.activemq.thread.DeterministicTaskRunner$1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.