You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Taras Tielkes (JIRA)" <ji...@apache.org> on 2010/03/23 23:33:46 UTC
[jira] Created: (AMQ-2660) OutOfMemoryError trying to consume big
message
OutOfMemoryError trying to consume big message
----------------------------------------------
Key: AMQ-2660
URL: https://issues.apache.org/activemq/browse/AMQ-2660
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.3.1
Reporter: Taras Tielkes
Running 5.3.1 broken on Win64, using Sun 1.6 x64 JVM.
Maximum heap size is set to 1024M.
We're sending and receiving huge messages over ActiveMQ, and cannot get the system running in a stable way.
For purposes of testing, we're sending text messages of 100M characters, I guess this means 200MB in-memory, since String are stored as UTF16 in JVM memory.
Testing with a single producer, single consumer.
We're using a slightly modified activemq.xml:
1) "systemUsage" block enabled, and configured as shown below:
-------
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="500 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
-------
2) producerFlowControl disabled, as shown below:
-------
<policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb">
-------
Description of symptoms:
1) Starting up producer, let it run until store gets full. This typically works out to 9 or 10 messages, each being 100M characters in size. From this I'm inferring that ActiveMQ stores these in UTF-8 format.
2) Starting up consumer
When starting the consumer, ActiveMQ broker fails with OOM shown below:
-------
ERROR | Failed to page in more queue messages
java.lang.OutOfMemoryError: Java heap space
at org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:681)
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:550)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:242)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:235)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:251)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:142)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1447)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1585)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1219)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
-------
Consumer keeps waiting, and never actually succeeds in consuming a message.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AMQ-2660) OutOfMemoryError trying to
consume big message
Posted by "Taras Tielkes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58433#action_58433 ]
Taras Tielkes edited comment on AMQ-2660 at 3/24/10 6:23 PM:
-------------------------------------------------------------
While this test uses a Java client, we're also using STOMP clients for other languages in production, so "useCompression" would not help there.
Apart from that, it's just the test that uses text messages. In production we transfer binary structures, which will benefit less from gzip compression.
I *do* appreciate the suggestion of using "useCompression" as a *workaround*, but I think there's a structural issue in the broker core here.
The broker should not hit an OOM while attempting to service a consumer queue read request.
was (Author: taras.tielkes):
While this test uses a Java client, we're also using STOMP clients for other languages in production, so "useCompression" would not help there.
Apart form that, it's just the test that uses text messages. In production we transfer binary structures, which will benefit less from gzip compression.
I *do* appreciate the suggestion of using "useCompression" as a *workaround*, but I think there's a structural issue in the broker core here.
The broker should not hit an OOM while attempting to service a consumer queue read request.
> OutOfMemoryError trying to consume big message
> ----------------------------------------------
>
> Key: AMQ-2660
> URL: https://issues.apache.org/activemq/browse/AMQ-2660
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.1
> Reporter: Taras Tielkes
>
> Running 5.3.1 broken on Win64, using Sun 1.6 x64 JVM.
> Maximum heap size is set to 1024M.
> We're sending and receiving huge messages over ActiveMQ, and cannot get the system running in a stable way.
> For purposes of testing, we're sending text messages of 100M characters, I guess this means 200MB in-memory, since Strings are stored as UTF16 in JVM memory.
> However, with the broker having a 1GB heap this should work - we assume.
> Testing with a single producer, single consumer.
> We're using a slightly modified activemq.xml:
> 1) "systemUsage" block enabled, and configured as shown below:
> -------
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="500 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> -------
> 2) producerFlowControl disabled, as shown below:
> -------
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb">
> -------
> Description of symptoms:
> 1) Starting up producer, let it run until store gets full. This typically works out to 9 or 10 messages, each being 100M characters in size. From this I'm inferring that ActiveMQ stores these in UTF-8 format.
> 2) Starting up consumer
> When starting the consumer, ActiveMQ broker fails with OOM shown below:
> -------
> ERROR | Failed to page in more queue messages
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:681)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:550)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:242)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:235)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:251)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:142)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1447)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1585)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1219)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> -------
> Consumer keeps waiting, and never actually succeeds in consuming a message.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2660) OutOfMemoryError trying to consume big
message
Posted by "Rob Davies (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58406#action_58406 ]
Rob Davies commented on AMQ-2660:
---------------------------------
You should try enabling compression - by setting the useCompression property on your ActiveMQConnectionFactory
> OutOfMemoryError trying to consume big message
> ----------------------------------------------
>
> Key: AMQ-2660
> URL: https://issues.apache.org/activemq/browse/AMQ-2660
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.1
> Reporter: Taras Tielkes
>
> Running 5.3.1 broken on Win64, using Sun 1.6 x64 JVM.
> Maximum heap size is set to 1024M.
> We're sending and receiving huge messages over ActiveMQ, and cannot get the system running in a stable way.
> For purposes of testing, we're sending text messages of 100M characters, I guess this means 200MB in-memory, since Strings are stored as UTF16 in JVM memory.
> However, with the broker having a 1GB heap this should work - we assume.
> Testing with a single producer, single consumer.
> We're using a slightly modified activemq.xml:
> 1) "systemUsage" block enabled, and configured as shown below:
> -------
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="500 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> -------
> 2) producerFlowControl disabled, as shown below:
> -------
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb">
> -------
> Description of symptoms:
> 1) Starting up producer, let it run until store gets full. This typically works out to 9 or 10 messages, each being 100M characters in size. From this I'm inferring that ActiveMQ stores these in UTF-8 format.
> 2) Starting up consumer
> When starting the consumer, ActiveMQ broker fails with OOM shown below:
> -------
> ERROR | Failed to page in more queue messages
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:681)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:550)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:242)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:235)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:251)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:142)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1447)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1585)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1219)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> -------
> Consumer keeps waiting, and never actually succeeds in consuming a message.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2660) OutOfMemoryError trying to consume big
message
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58434#action_58434 ]
Gary Tully commented on AMQ-2660:
---------------------------------
if you have really large messages, they the destination pageSize can be reduced and there is also lazyDispatch that with a prefetch of 1 would ensure that there are only as many messages as concurrent consumers in memory at a time.
> OutOfMemoryError trying to consume big message
> ----------------------------------------------
>
> Key: AMQ-2660
> URL: https://issues.apache.org/activemq/browse/AMQ-2660
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.1
> Reporter: Taras Tielkes
>
> Running 5.3.1 broken on Win64, using Sun 1.6 x64 JVM.
> Maximum heap size is set to 1024M.
> We're sending and receiving huge messages over ActiveMQ, and cannot get the system running in a stable way.
> For purposes of testing, we're sending text messages of 100M characters, I guess this means 200MB in-memory, since Strings are stored as UTF16 in JVM memory.
> However, with the broker having a 1GB heap this should work - we assume.
> Testing with a single producer, single consumer.
> We're using a slightly modified activemq.xml:
> 1) "systemUsage" block enabled, and configured as shown below:
> -------
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="500 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> -------
> 2) producerFlowControl disabled, as shown below:
> -------
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb">
> -------
> Description of symptoms:
> 1) Starting up producer, let it run until store gets full. This typically works out to 9 or 10 messages, each being 100M characters in size. From this I'm inferring that ActiveMQ stores these in UTF-8 format.
> 2) Starting up consumer
> When starting the consumer, ActiveMQ broker fails with OOM shown below:
> -------
> ERROR | Failed to page in more queue messages
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:681)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:550)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:242)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:235)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:251)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:142)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1447)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1585)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1219)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> -------
> Consumer keeps waiting, and never actually succeeds in consuming a message.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2660) OutOfMemoryError trying to consume big
message
Posted by "Bruce Snyder (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bruce Snyder updated AMQ-2660:
------------------------------
Fix Version/s: 5.5.0
(was: 5.4.1)
> OutOfMemoryError trying to consume big message
> ----------------------------------------------
>
> Key: AMQ-2660
> URL: https://issues.apache.org/activemq/browse/AMQ-2660
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.1
> Reporter: Taras Tielkes
> Fix For: 5.5.0
>
>
> Running 5.3.1 broken on Win64, using Sun 1.6 x64 JVM.
> Maximum heap size is set to 1024M.
> We're sending and receiving huge messages over ActiveMQ, and cannot get the system running in a stable way.
> For purposes of testing, we're sending text messages of 100M characters, I guess this means 200MB in-memory, since Strings are stored as UTF16 in JVM memory.
> However, with the broker having a 1GB heap this should work - we assume.
> Testing with a single producer, single consumer.
> We're using a slightly modified activemq.xml:
> 1) "systemUsage" block enabled, and configured as shown below:
> -------
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="500 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> -------
> 2) producerFlowControl disabled, as shown below:
> -------
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb">
> -------
> Description of symptoms:
> 1) Starting up producer, let it run until store gets full. This typically works out to 9 or 10 messages, each being 100M characters in size. From this I'm inferring that ActiveMQ stores these in UTF-8 format.
> 2) Starting up consumer
> When starting the consumer, ActiveMQ broker fails with OOM shown below:
> -------
> ERROR | Failed to page in more queue messages
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:681)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:550)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:242)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:235)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:251)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:142)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1447)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1585)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1219)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> -------
> Consumer keeps waiting, and never actually succeeds in consuming a message.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2660) OutOfMemoryError trying to consume big
message
Posted by "Taras Tielkes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58433#action_58433 ]
Taras Tielkes commented on AMQ-2660:
------------------------------------
While this test uses a Java client, we're also using STOMP clients for other languages in production, so "useCompression" would not help there.
Apart form that, it's just the test that uses text messages. In production we transfer binary structures, which will benefit less from gzip compression.
I *do* appreciate the suggestion of using "useCompression" as a *workaround*, but I think there's a structural issue in the broker core here.
The broker should not hit an OOM while attempting to service a consumer queue read request.
> OutOfMemoryError trying to consume big message
> ----------------------------------------------
>
> Key: AMQ-2660
> URL: https://issues.apache.org/activemq/browse/AMQ-2660
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.1
> Reporter: Taras Tielkes
>
> Running 5.3.1 broken on Win64, using Sun 1.6 x64 JVM.
> Maximum heap size is set to 1024M.
> We're sending and receiving huge messages over ActiveMQ, and cannot get the system running in a stable way.
> For purposes of testing, we're sending text messages of 100M characters, I guess this means 200MB in-memory, since Strings are stored as UTF16 in JVM memory.
> However, with the broker having a 1GB heap this should work - we assume.
> Testing with a single producer, single consumer.
> We're using a slightly modified activemq.xml:
> 1) "systemUsage" block enabled, and configured as shown below:
> -------
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="500 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> -------
> 2) producerFlowControl disabled, as shown below:
> -------
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb">
> -------
> Description of symptoms:
> 1) Starting up producer, let it run until store gets full. This typically works out to 9 or 10 messages, each being 100M characters in size. From this I'm inferring that ActiveMQ stores these in UTF-8 format.
> 2) Starting up consumer
> When starting the consumer, ActiveMQ broker fails with OOM shown below:
> -------
> ERROR | Failed to page in more queue messages
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:681)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:550)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:242)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:235)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:251)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:142)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1447)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1585)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1219)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> -------
> Consumer keeps waiting, and never actually succeeds in consuming a message.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2660) OutOfMemoryError trying to consume big
message
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58445#action_58445 ]
Gary Tully commented on AMQ-2660:
---------------------------------
the details of how to configure are in http://activemq.apache.org/per-destination-policies.html
The attributes in question are maxPageSize (reduce from the 200 default value) or queuePrefetch and lazyDispatch.
> OutOfMemoryError trying to consume big message
> ----------------------------------------------
>
> Key: AMQ-2660
> URL: https://issues.apache.org/activemq/browse/AMQ-2660
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.1
> Reporter: Taras Tielkes
>
> Running 5.3.1 broken on Win64, using Sun 1.6 x64 JVM.
> Maximum heap size is set to 1024M.
> We're sending and receiving huge messages over ActiveMQ, and cannot get the system running in a stable way.
> For purposes of testing, we're sending text messages of 100M characters, I guess this means 200MB in-memory, since Strings are stored as UTF16 in JVM memory.
> However, with the broker having a 1GB heap this should work - we assume.
> Testing with a single producer, single consumer.
> We're using a slightly modified activemq.xml:
> 1) "systemUsage" block enabled, and configured as shown below:
> -------
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="500 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> -------
> 2) producerFlowControl disabled, as shown below:
> -------
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb">
> -------
> Description of symptoms:
> 1) Starting up producer, let it run until store gets full. This typically works out to 9 or 10 messages, each being 100M characters in size. From this I'm inferring that ActiveMQ stores these in UTF-8 format.
> 2) Starting up consumer
> When starting the consumer, ActiveMQ broker fails with OOM shown below:
> -------
> ERROR | Failed to page in more queue messages
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:681)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:550)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:242)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:235)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:251)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:142)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1447)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1585)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1219)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> -------
> Consumer keeps waiting, and never actually succeeds in consuming a message.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2660) OutOfMemoryError trying to consume big
message
Posted by "Taras Tielkes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Taras Tielkes updated AMQ-2660:
-------------------------------
Description:
Running 5.3.1 broken on Win64, using Sun 1.6 x64 JVM.
Maximum heap size is set to 1024M.
We're sending and receiving huge messages over ActiveMQ, and cannot get the system running in a stable way.
For purposes of testing, we're sending text messages of 100M characters, I guess this means 200MB in-memory, since Strings are stored as UTF16 in JVM memory.
However, with the broker having a 1GB heap this should work - we assume.
Testing with a single producer, single consumer.
We're using a slightly modified activemq.xml:
1) "systemUsage" block enabled, and configured as shown below:
-------
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="500 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
-------
2) producerFlowControl disabled, as shown below:
-------
<policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb">
-------
Description of symptoms:
1) Starting up producer, let it run until store gets full. This typically works out to 9 or 10 messages, each being 100M characters in size. From this I'm inferring that ActiveMQ stores these in UTF-8 format.
2) Starting up consumer
When starting the consumer, ActiveMQ broker fails with OOM shown below:
-------
ERROR | Failed to page in more queue messages
java.lang.OutOfMemoryError: Java heap space
at org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:681)
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:550)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:242)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:235)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:251)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:142)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1447)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1585)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1219)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
-------
Consumer keeps waiting, and never actually succeeds in consuming a message.
was:
Running 5.3.1 broken on Win64, using Sun 1.6 x64 JVM.
Maximum heap size is set to 1024M.
We're sending and receiving huge messages over ActiveMQ, and cannot get the system running in a stable way.
For purposes of testing, we're sending text messages of 100M characters, I guess this means 200MB in-memory, since String are stored as UTF16 in JVM memory.
Testing with a single producer, single consumer.
We're using a slightly modified activemq.xml:
1) "systemUsage" block enabled, and configured as shown below:
-------
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="500 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
-------
2) producerFlowControl disabled, as shown below:
-------
<policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb">
-------
Description of symptoms:
1) Starting up producer, let it run until store gets full. This typically works out to 9 or 10 messages, each being 100M characters in size. From this I'm inferring that ActiveMQ stores these in UTF-8 format.
2) Starting up consumer
When starting the consumer, ActiveMQ broker fails with OOM shown below:
-------
ERROR | Failed to page in more queue messages
java.lang.OutOfMemoryError: Java heap space
at org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:681)
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:550)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:242)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:235)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:251)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:142)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1447)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1585)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1219)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
-------
Consumer keeps waiting, and never actually succeeds in consuming a message.
> OutOfMemoryError trying to consume big message
> ----------------------------------------------
>
> Key: AMQ-2660
> URL: https://issues.apache.org/activemq/browse/AMQ-2660
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.1
> Reporter: Taras Tielkes
>
> Running 5.3.1 broken on Win64, using Sun 1.6 x64 JVM.
> Maximum heap size is set to 1024M.
> We're sending and receiving huge messages over ActiveMQ, and cannot get the system running in a stable way.
> For purposes of testing, we're sending text messages of 100M characters, I guess this means 200MB in-memory, since Strings are stored as UTF16 in JVM memory.
> However, with the broker having a 1GB heap this should work - we assume.
> Testing with a single producer, single consumer.
> We're using a slightly modified activemq.xml:
> 1) "systemUsage" block enabled, and configured as shown below:
> -------
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="500 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> -------
> 2) producerFlowControl disabled, as shown below:
> -------
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb">
> -------
> Description of symptoms:
> 1) Starting up producer, let it run until store gets full. This typically works out to 9 or 10 messages, each being 100M characters in size. From this I'm inferring that ActiveMQ stores these in UTF-8 format.
> 2) Starting up consumer
> When starting the consumer, ActiveMQ broker fails with OOM shown below:
> -------
> ERROR | Failed to page in more queue messages
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:681)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:550)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:242)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:235)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:251)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:142)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1447)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1585)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1219)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> -------
> Consumer keeps waiting, and never actually succeeds in consuming a message.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2660) OutOfMemoryError trying to consume big
message
Posted by "Taras Tielkes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58438#action_58438 ]
Taras Tielkes commented on AMQ-2660:
------------------------------------
Gary, could you specific in a bit more details which elements/attributes I need to add to make it work like you suggest?
I'd be happy to perform a batch of stress tests in our environment based on your suggestions.
> OutOfMemoryError trying to consume big message
> ----------------------------------------------
>
> Key: AMQ-2660
> URL: https://issues.apache.org/activemq/browse/AMQ-2660
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.1
> Reporter: Taras Tielkes
>
> Running 5.3.1 broken on Win64, using Sun 1.6 x64 JVM.
> Maximum heap size is set to 1024M.
> We're sending and receiving huge messages over ActiveMQ, and cannot get the system running in a stable way.
> For purposes of testing, we're sending text messages of 100M characters, I guess this means 200MB in-memory, since Strings are stored as UTF16 in JVM memory.
> However, with the broker having a 1GB heap this should work - we assume.
> Testing with a single producer, single consumer.
> We're using a slightly modified activemq.xml:
> 1) "systemUsage" block enabled, and configured as shown below:
> -------
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="500 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> -------
> 2) producerFlowControl disabled, as shown below:
> -------
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb">
> -------
> Description of symptoms:
> 1) Starting up producer, let it run until store gets full. This typically works out to 9 or 10 messages, each being 100M characters in size. From this I'm inferring that ActiveMQ stores these in UTF-8 format.
> 2) Starting up consumer
> When starting the consumer, ActiveMQ broker fails with OOM shown below:
> -------
> ERROR | Failed to page in more queue messages
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:681)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:550)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:242)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:235)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:251)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:142)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1447)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1585)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1219)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> -------
> Consumer keeps waiting, and never actually succeeds in consuming a message.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.