You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@activemq.apache.org by "Lionel Cons (JIRA)" <ji...@apache.org> on 2012/09/04 08:15:07 UTC
[jira] [Created] (APLO-257) Better memory usage (avoid Java OOM
exceptions)
Lionel Cons created APLO-257:
--------------------------------
Summary: Better memory usage (avoid Java OOM exceptions)
Key: APLO-257
URL: https://issues.apache.org/jira/browse/APLO-257
Project: ActiveMQ Apollo
Issue Type: Improvement
Environment: apollo-99-trunk-20120902.143447-110
Reporter: Lionel Cons
When running a stomp-benchmark based stress test against Apollo, it generated several OOM exceptions like:
2012-09-03 23:51:45,275 | WARN | handle failed |
java.lang.OutOfMemoryError: Java heap space
2012-09-03 23:52:02,428 | WARN | |
java.lang.OutOfMemoryError: Java heap space
2012-09-03 23:52:01,372 | WARN | Error for /broker |
java.lang.OutOfMemoryError: Java heap space
2012-09-03 23:51:55,554 | WARN | Problem scavenging sessions |
java.lang.OutOfMemoryError: Java heap space
The JVM already had 8GB of heap. FWIW, here are the options used: "-server -Xmx8192m -Xms4096m -XX:-UseBiasedLocking".
Would it be possible for Apollo to detect when its getting close to the JVM limits and adjust its memory usage accordingly (reducing buffers, swapping messages to disk...)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (APLO-257) Better memory usage (avoid Java OOM
exceptions)
Posted by "Lionel Cons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500880#comment-13500880 ]
Lionel Cons commented on APLO-257:
----------------------------------
FWIW, we tried once more with a newer snapshot (20121114-223000-129) and got the exact same behavior: several OOM exceptions:
2012-11-19 20:51:18,363 | WARN | java.lang.OutOfMemoryError: GC overhead limit exceeded | 13b1928a88b
2012-11-19 20:51:18,364 | WARN | DB operation failed. (entering recovery mode): java.lang.OutOfMemoryError: GC overhead limit exceeded | 13b1928a884
2012-11-19 20:51:18,364 | WARN | DB operation failed. (entering recovery mode): java.lang.OutOfMemoryError: GC overhead limit exceeded | 13b1928a885
2012-11-19 20:51:18,364 | WARN | java.lang.OutOfMemoryError: GC overhead limit exceeded | 13b1928a888
2012-11-19 20:51:18,363 | WARN | java.lang.OutOfMemoryError: GC overhead limit exceeded | 13b1928a889
2012-11-19 20:51:18,363 | WARN | java.lang.OutOfMemoryError: GC overhead limit exceeded | 13b1928a887
We would appreciate progress on this issue.
> Better memory usage (avoid Java OOM exceptions)
> -----------------------------------------------
>
> Key: APLO-257
> URL: https://issues.apache.org/jira/browse/APLO-257
> Project: ActiveMQ Apollo
> Issue Type: Improvement
> Environment: apollo-99-trunk-20120902.143447-110
> Reporter: Lionel Cons
>
> When running a stomp-benchmark based stress test against Apollo, it generated several OOM exceptions like:
> 2012-09-03 23:51:45,275 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:02,428 | WARN | |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:01,372 | WARN | Error for /broker |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:51:55,554 | WARN | Problem scavenging sessions |
> java.lang.OutOfMemoryError: Java heap space
> The JVM already had 8GB of heap. FWIW, here are the options used: "-server -Xmx8192m -Xms4096m -XX:-UseBiasedLocking".
> Would it be possible for Apollo to detect when its getting close to the JVM limits and adjust its memory usage accordingly (reducing buffers, swapping messages to disk...)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (APLO-257) Better memory usage (avoid Java OOM
exceptions)
Posted by "Christian Posta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453982#comment-13453982 ]
Christian Posta commented on APLO-257:
--------------------------------------
Lionel,
Is this something that happens regularly?
Any other hints (machine specs, etc) for how to create this?
I've been trying and cannot recreate it yet (using the stomp benchmark from APLO-176)
> Better memory usage (avoid Java OOM exceptions)
> -----------------------------------------------
>
> Key: APLO-257
> URL: https://issues.apache.org/jira/browse/APLO-257
> Project: ActiveMQ Apollo
> Issue Type: Improvement
> Environment: apollo-99-trunk-20120902.143447-110
> Reporter: Lionel Cons
>
> When running a stomp-benchmark based stress test against Apollo, it generated several OOM exceptions like:
> 2012-09-03 23:51:45,275 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:02,428 | WARN | |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:01,372 | WARN | Error for /broker |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:51:55,554 | WARN | Problem scavenging sessions |
> java.lang.OutOfMemoryError: Java heap space
> The JVM already had 8GB of heap. FWIW, here are the options used: "-server -Xmx8192m -Xms4096m -XX:-UseBiasedLocking".
> Would it be possible for Apollo to detect when its getting close to the JVM limits and adjust its memory usage accordingly (reducing buffers, swapping messages to disk...)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (APLO-257) Better memory usage (avoid Java OOM
exceptions)
Posted by "Lionel Cons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467502#comment-13467502 ]
Lionel Cons commented on APLO-257:
----------------------------------
We ran the same benchmark scenario on the same hardware with the latest snapshot (apollo-99-trunk-20120929.031419-121) with two changes:
- more memory: -server -Xmx16384m -Xms4096m -XX:-UseBiasedLocking
- using the default slow_consumer_policy="block"
We still see memory exceptions, however, these are "GC overhead limit exceeded". Here are some stack traces:
2012-10-01 18:49:51,934 | WARN | stackref=13a1c2bf7a8
java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.fusesource.hawtdispatch.internal.util.TimerHeap.executeReadyTimers(TimerHeap.java:79)
at org.fusesource.hawtdispatch.internal.TimerThread.run(TimerThread.java:144)
2012-10-01 18:50:14,710 | WARN | stackref=13a1c2bf7a9
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:671)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:721)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at org.fusesource.hawtdispatch.package$RichExecutor.execute(hawtdispatch.scala:171)
at org.fusesource.hawtdispatch.package$RichExecutorTrait$class.apply(hawtdispatch.scala:68)
at org.fusesource.hawtdispatch.package$RichExecutor.apply(hawtdispatch.scala:169)
at org.apache.activemq.apollo.broker.security.JaasAuthenticator.authenticate(JaasAuthenticator.scala:66)
at org.apache.activemq.apollo.stomp.StompProtocolHandler$$anonfun$on_stomp_connect$3$$anonfun$apply$mcV$sp$1.apply$mcV$sp(StompProtocolHandler.scala:1107)
at org.fusesource.hawtdispatch.package$$anon$3.run(hawtdispatch.scala:357)
at org.fusesource.hawtdispatch.internal.SerialDispatchQueue.run(SerialDispatchQueue.java:96)
at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:77)
2012-10-01 18:50:48,827 | WARN | stackref=13a1c2bf7aa
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3065)
at org.fusesource.hawtdispatch.transport.AbstractProtocolCodec.read(AbstractProtocolCodec.java:307)
at org.fusesource.hawtdispatch.transport.TcpTransport.drainInbound(TcpTransport.java:657)
at org.fusesource.hawtdispatch.transport.TcpTransport$6.run(TcpTransport.java:538)
at org.fusesource.hawtdispatch.internal.NioDispatchSource$3.run(NioDispatchSource.java:226)
at org.fusesource.hawtdispatch.internal.SerialDispatchQueue.run(SerialDispatchQueue.java:96)
at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:77)
2012-10-01 18:51:08,070 | WARN | stackref=13a1c2bf7ab
java.lang.OutOfMemoryError: GC overhead limit exceeded
2012-10-01 18:51:38,771 | WARN | stackref=13a1c2bf7ac
java.lang.OutOfMemoryError: GC overhead limit exceeded
It would be good if Apollo could run happily in its 16GB without causing memory related exceptions...
> Better memory usage (avoid Java OOM exceptions)
> -----------------------------------------------
>
> Key: APLO-257
> URL: https://issues.apache.org/jira/browse/APLO-257
> Project: ActiveMQ Apollo
> Issue Type: Improvement
> Environment: apollo-99-trunk-20120902.143447-110
> Reporter: Lionel Cons
>
> When running a stomp-benchmark based stress test against Apollo, it generated several OOM exceptions like:
> 2012-09-03 23:51:45,275 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:02,428 | WARN | |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:01,372 | WARN | Error for /broker |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:51:55,554 | WARN | Problem scavenging sessions |
> java.lang.OutOfMemoryError: Java heap space
> The JVM already had 8GB of heap. FWIW, here are the options used: "-server -Xmx8192m -Xms4096m -XX:-UseBiasedLocking".
> Would it be possible for Apollo to detect when its getting close to the JVM limits and adjust its memory usage accordingly (reducing buffers, swapping messages to disk...)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (APLO-257) Better memory usage (avoid Java OOM
exceptions)
Posted by "Christian Posta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505879#comment-13505879 ]
Christian Posta commented on APLO-257:
--------------------------------------
Lionel,
I recently got access to some larger boxes like yours. I'll give it another go when I get a sec. I'll keep you updated...
> Better memory usage (avoid Java OOM exceptions)
> -----------------------------------------------
>
> Key: APLO-257
> URL: https://issues.apache.org/jira/browse/APLO-257
> Project: ActiveMQ Apollo
> Issue Type: Improvement
> Environment: apollo-99-trunk-20120902.143447-110
> Reporter: Lionel Cons
>
> When running a stomp-benchmark based stress test against Apollo, it generated several OOM exceptions like:
> 2012-09-03 23:51:45,275 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:02,428 | WARN | |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:01,372 | WARN | Error for /broker |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:51:55,554 | WARN | Problem scavenging sessions |
> java.lang.OutOfMemoryError: Java heap space
> The JVM already had 8GB of heap. FWIW, here are the options used: "-server -Xmx8192m -Xms4096m -XX:-UseBiasedLocking".
> Would it be possible for Apollo to detect when its getting close to the JVM limits and adjust its memory usage accordingly (reducing buffers, swapping messages to disk...)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (APLO-257) Better memory usage (avoid Java OOM
exceptions)
Posted by "Lionel Cons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454655#comment-13454655 ]
Lionel Cons commented on APLO-257:
----------------------------------
Christian,
We've seen this twice so far.
Our testing machines (the one running stomp-benchmark and the one running Apollo) are identical (GenuineIntel: Intel(R) Xeon(R) CPU L5520 @ 2.27GHz; 2 CPU(s) x 4 core(s) (hyperthreading: on) = 16 logical CPU(s)) and connected via a 10Gb link.
Cheers,
Lionel
> Better memory usage (avoid Java OOM exceptions)
> -----------------------------------------------
>
> Key: APLO-257
> URL: https://issues.apache.org/jira/browse/APLO-257
> Project: ActiveMQ Apollo
> Issue Type: Improvement
> Environment: apollo-99-trunk-20120902.143447-110
> Reporter: Lionel Cons
>
> When running a stomp-benchmark based stress test against Apollo, it generated several OOM exceptions like:
> 2012-09-03 23:51:45,275 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:02,428 | WARN | |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:01,372 | WARN | Error for /broker |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:51:55,554 | WARN | Problem scavenging sessions |
> java.lang.OutOfMemoryError: Java heap space
> The JVM already had 8GB of heap. FWIW, here are the options used: "-server -Xmx8192m -Xms4096m -XX:-UseBiasedLocking".
> Would it be possible for Apollo to detect when its getting close to the JVM limits and adjust its memory usage accordingly (reducing buffers, swapping messages to disk...)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (APLO-257) Better memory usage (avoid Java OOM
exceptions)
Posted by "Lionel Cons (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/APLO-257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447506#comment-13447506 ]
Lionel Cons commented on APLO-257:
----------------------------------
FWIW, the stomp-benchmark scenario file used is the one attached to APLO-176. So only "reasonable" stress, without huge messages or a very high number of concurrent connections.
Also, the apollo.xml file contains default settings except slow_consumer_policy="queue" for topics, which should not impact memory usage too much since pending messages should be sent to queue so they should eventually be swapped out to disk.
> Better memory usage (avoid Java OOM exceptions)
> -----------------------------------------------
>
> Key: APLO-257
> URL: https://issues.apache.org/jira/browse/APLO-257
> Project: ActiveMQ Apollo
> Issue Type: Improvement
> Environment: apollo-99-trunk-20120902.143447-110
> Reporter: Lionel Cons
>
> When running a stomp-benchmark based stress test against Apollo, it generated several OOM exceptions like:
> 2012-09-03 23:51:45,275 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:02,428 | WARN | |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:01,372 | WARN | Error for /broker |
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:51:55,554 | WARN | Problem scavenging sessions |
> java.lang.OutOfMemoryError: Java heap space
> The JVM already had 8GB of heap. FWIW, here are the options used: "-server -Xmx8192m -Xms4096m -XX:-UseBiasedLocking".
> Would it be possible for Apollo to detect when its getting close to the JVM limits and adjust its memory usage accordingly (reducing buffers, swapping messages to disk...)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira