You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Robert Voliva (JIRA)" <ji...@apache.org> on 2012/07/16 19:56:35 UTC
[jira] [Created] (AMQ-3931) Memory problems with large transactions
Robert Voliva created AMQ-3931:
----------------------------------
Summary: Memory problems with large transactions
Key: AMQ-3931
URL: https://issues.apache.org/jira/browse/AMQ-3931
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.6.0
Environment: OS: RHEL 5.5
Startup options: -Xmx2G -Xms2G -XX:MaxPermSize=512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dactivemq.classpath=/opt/activemq/conf; -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Dactivemq.conf=/opt/activemq/conf -Dactivemq.data=/opt/activemq/data -Djava.io.tmpdir=/opt/activemq/tmp
Config file: activemq.xml attached
Reporter: Robert Voliva
Attachments: activemq_log_output.txt
When running a single transaction that produces 300,000 text messages, ActiveMQ runs into all sorts of memory problems. The log output is attached. We were using a simple Java JMS client class to produce the 300,000 messages in a single transaction. Each message's text payload was a random string of 10,000 bytes.
This is a production scenario that we have that's killing our broker. We isolated it down to its most basic pieces to try to test where the breakdown was occuring, which led us to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3931) Memory problems with large
transactions
Posted by "Robert Voliva (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415624#comment-13415624 ]
Robert Voliva commented on AMQ-3931:
------------------------------------
Do you know if this is a configuration option of ActiveMQ, where it can be told to serialize messages participating in a transaction to disk while more messages for that transaction are coming in? This would allow it to free up memory for the rest of the incoming messages. Or is this just a feature/limitation in how ActiveMQ works?
> Memory problems with large transactions
> ---------------------------------------
>
> Key: AMQ-3931
> URL: https://issues.apache.org/jira/browse/AMQ-3931
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Environment: OS: RHEL 5.5
> Startup options: -Xmx2G -Xms2G -XX:MaxPermSize=512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dactivemq.classpath=/opt/activemq/conf; -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Dactivemq.conf=/opt/activemq/conf -Dactivemq.data=/opt/activemq/data -Djava.io.tmpdir=/opt/activemq/tmp
> Config file: activemq.xml attached
> Reporter: Robert Voliva
> Attachments: activemq_log_output.txt, gbsmq.xml
>
>
> When running a single transaction that produces 300,000 text messages, ActiveMQ runs into all sorts of memory problems. The log output is attached. We were using a simple Java JMS client class to produce the 300,000 messages in a single transaction. Each message's text payload was a random string of 10,000 bytes.
> This is a production scenario that we have that's killing our broker. We isolated it down to its most basic pieces to try to test where the breakdown was occuring, which led us to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3931) Memory problems with large
transactions
Posted by "Fengming Lou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415602#comment-13415602 ]
Fengming Lou commented on AMQ-3931:
-----------------------------------
you have in single transaction of 300,000 text messages with average size of ~10,000 bytes. If they are all in memory at the same time, it would require 300,000 x 10,000 = 3G. You have only 1 g allocated for the broker?
<memoryUsage>
<memoryUsage limit="1 gb"/>
</memoryUsage>
> Memory problems with large transactions
> ---------------------------------------
>
> Key: AMQ-3931
> URL: https://issues.apache.org/jira/browse/AMQ-3931
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Environment: OS: RHEL 5.5
> Startup options: -Xmx2G -Xms2G -XX:MaxPermSize=512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dactivemq.classpath=/opt/activemq/conf; -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Dactivemq.conf=/opt/activemq/conf -Dactivemq.data=/opt/activemq/data -Djava.io.tmpdir=/opt/activemq/tmp
> Config file: activemq.xml attached
> Reporter: Robert Voliva
> Attachments: activemq_log_output.txt, gbsmq.xml
>
>
> When running a single transaction that produces 300,000 text messages, ActiveMQ runs into all sorts of memory problems. The log output is attached. We were using a simple Java JMS client class to produce the 300,000 messages in a single transaction. Each message's text payload was a random string of 10,000 bytes.
> This is a production scenario that we have that's killing our broker. We isolated it down to its most basic pieces to try to test where the breakdown was occuring, which led us to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3931) Memory problems with large
transactions
Posted by "Robert Voliva (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416568#comment-13416568 ]
Robert Voliva commented on AMQ-3931:
------------------------------------
Thanks Fengming, I went through all that documentation and did make some changes. What we're still seeing is that when producing even 100,000 messages in a single transaction, they all seem to be kept in memory until (presumably) the transaction is committed. I have verified that producing the same number of messages in multiple transactions, it behaves as expected, and doesn't kill the broker. It is able to GC periodically and things hum along great.
This seems to all be related to producing that many messages in one transaction.
> Memory problems with large transactions
> ---------------------------------------
>
> Key: AMQ-3931
> URL: https://issues.apache.org/jira/browse/AMQ-3931
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Environment: OS: RHEL 5.5
> Startup options: -Xmx2G -Xms2G -XX:MaxPermSize=512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dactivemq.classpath=/opt/activemq/conf; -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Dactivemq.conf=/opt/activemq/conf -Dactivemq.data=/opt/activemq/data -Djava.io.tmpdir=/opt/activemq/tmp
> Config file: activemq.xml attached
> Reporter: Robert Voliva
> Attachments: activemq_log_output.txt, gbsmq.xml
>
>
> When running a single transaction that produces 300,000 text messages, ActiveMQ runs into all sorts of memory problems. The log output is attached. We were using a simple Java JMS client class to produce the 300,000 messages in a single transaction. Each message's text payload was a random string of 10,000 bytes.
> This is a production scenario that we have that's killing our broker. We isolated it down to its most basic pieces to try to test where the breakdown was occuring, which led us to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (AMQ-3931) Memory problems with large
transactions
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved AMQ-3931.
------------------------------
Resolution: Not A Problem
Fix Version/s: 5.7.0
Assignee: Claus Ibsen
This is working as designed. pending messages in a TX is kept in memory.
As you have excessive number of messages, make sure your broker have sufficient memory to deal with this requirement.
> Memory problems with large transactions
> ---------------------------------------
>
> Key: AMQ-3931
> URL: https://issues.apache.org/jira/browse/AMQ-3931
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Environment: OS: RHEL 5.5
> Startup options: -Xmx2G -Xms2G -XX:MaxPermSize=512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dactivemq.classpath=/opt/activemq/conf; -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Dactivemq.conf=/opt/activemq/conf -Dactivemq.data=/opt/activemq/data -Djava.io.tmpdir=/opt/activemq/tmp
> Config file: activemq.xml attached
> Reporter: Robert Voliva
> Assignee: Claus Ibsen
> Fix For: 5.7.0
>
> Attachments: activemq_log_output.txt, gbsmq.xml
>
>
> When running a single transaction that produces 300,000 text messages, ActiveMQ runs into all sorts of memory problems. The log output is attached. We were using a simple Java JMS client class to produce the 300,000 messages in a single transaction. Each message's text payload was a random string of 10,000 bytes.
> This is a production scenario that we have that's killing our broker. We isolated it down to its most basic pieces to try to test where the breakdown was occuring, which led us to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3931) Memory problems with large
transactions
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458617#comment-13458617 ]
Claus Ibsen commented on AMQ-3931:
----------------------------------
See also this ticket. KahaDB has a special option for overflow to disk for TX
https://issues.apache.org/jira/browse/AMQ-3374
> Memory problems with large transactions
> ---------------------------------------
>
> Key: AMQ-3931
> URL: https://issues.apache.org/jira/browse/AMQ-3931
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Environment: OS: RHEL 5.5
> Startup options: -Xmx2G -Xms2G -XX:MaxPermSize=512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dactivemq.classpath=/opt/activemq/conf; -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Dactivemq.conf=/opt/activemq/conf -Dactivemq.data=/opt/activemq/data -Djava.io.tmpdir=/opt/activemq/tmp
> Config file: activemq.xml attached
> Reporter: Robert Voliva
> Assignee: Claus Ibsen
> Fix For: 5.7.0
>
> Attachments: activemq_log_output.txt, gbsmq.xml
>
>
> When running a single transaction that produces 300,000 text messages, ActiveMQ runs into all sorts of memory problems. The log output is attached. We were using a simple Java JMS client class to produce the 300,000 messages in a single transaction. Each message's text payload was a random string of 10,000 bytes.
> This is a production scenario that we have that's killing our broker. We isolated it down to its most basic pieces to try to test where the breakdown was occuring, which led us to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3931) Memory problems with large transactions
Posted by "Robert Voliva (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Voliva updated AMQ-3931:
-------------------------------
Attachment: gbsmq.xml
> Memory problems with large transactions
> ---------------------------------------
>
> Key: AMQ-3931
> URL: https://issues.apache.org/jira/browse/AMQ-3931
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Environment: OS: RHEL 5.5
> Startup options: -Xmx2G -Xms2G -XX:MaxPermSize=512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dactivemq.classpath=/opt/activemq/conf; -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Dactivemq.conf=/opt/activemq/conf -Dactivemq.data=/opt/activemq/data -Djava.io.tmpdir=/opt/activemq/tmp
> Config file: activemq.xml attached
> Reporter: Robert Voliva
> Attachments: activemq_log_output.txt, gbsmq.xml
>
>
> When running a single transaction that produces 300,000 text messages, ActiveMQ runs into all sorts of memory problems. The log output is attached. We were using a simple Java JMS client class to produce the 300,000 messages in a single transaction. Each message's text payload was a random string of 10,000 bytes.
> This is a production scenario that we have that's killing our broker. We isolated it down to its most basic pieces to try to test where the breakdown was occuring, which led us to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3931) Memory problems with large transactions
Posted by "Robert Voliva (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Voliva updated AMQ-3931:
-------------------------------
Attachment: activemq_log_output.txt
> Memory problems with large transactions
> ---------------------------------------
>
> Key: AMQ-3931
> URL: https://issues.apache.org/jira/browse/AMQ-3931
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Environment: OS: RHEL 5.5
> Startup options: -Xmx2G -Xms2G -XX:MaxPermSize=512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dactivemq.classpath=/opt/activemq/conf; -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Dactivemq.conf=/opt/activemq/conf -Dactivemq.data=/opt/activemq/data -Djava.io.tmpdir=/opt/activemq/tmp
> Config file: activemq.xml attached
> Reporter: Robert Voliva
> Attachments: activemq_log_output.txt
>
>
> When running a single transaction that produces 300,000 text messages, ActiveMQ runs into all sorts of memory problems. The log output is attached. We were using a simple Java JMS client class to produce the 300,000 messages in a single transaction. Each message's text payload was a random string of 10,000 bytes.
> This is a production scenario that we have that's killing our broker. We isolated it down to its most basic pieces to try to test where the breakdown was occuring, which led us to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3931) Memory problems with large
transactions
Posted by "Fengming Lou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415654#comment-13415654 ]
Fengming Lou commented on AMQ-3931:
-----------------------------------
Following links should give shine some light on your issue. I see a lot of queues in your configurations, but not memoryLimit imposed on any queues.
http://activemq.apache.org/javalangoutofmemory.html
http://activemq.apache.org/per-destination-policies.html
http://activemq.apache.org/message-cursors.html
> Memory problems with large transactions
> ---------------------------------------
>
> Key: AMQ-3931
> URL: https://issues.apache.org/jira/browse/AMQ-3931
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Environment: OS: RHEL 5.5
> Startup options: -Xmx2G -Xms2G -XX:MaxPermSize=512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dactivemq.classpath=/opt/activemq/conf; -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Dactivemq.conf=/opt/activemq/conf -Dactivemq.data=/opt/activemq/data -Djava.io.tmpdir=/opt/activemq/tmp
> Config file: activemq.xml attached
> Reporter: Robert Voliva
> Attachments: activemq_log_output.txt, gbsmq.xml
>
>
> When running a single transaction that produces 300,000 text messages, ActiveMQ runs into all sorts of memory problems. The log output is attached. We were using a simple Java JMS client class to produce the 300,000 messages in a single transaction. Each message's text payload was a random string of 10,000 bytes.
> This is a production scenario that we have that's killing our broker. We isolated it down to its most basic pieces to try to test where the breakdown was occuring, which led us to this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira