You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@activemq.apache.org by "Daniel Miles (JIRA)" <ji...@apache.org> on 2014/06/26 18:39:24 UTC
[jira] [Commented] (APLO-360) regular, reproducable OutOfMemory
exceptions
[ https://issues.apache.org/jira/browse/APLO-360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044830#comment-14044830 ]
Daniel Miles commented on APLO-360:
-----------------------------------
Oh, one more thing, here's my apollo config (my load test was using a queue called "fourth":
{code}
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version
2.0 (the "License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0 Unless required by
applicable law or agreed to in writing, software distributed under
the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and
limitations under the License.
-->
<!--
For more information on how configure this file please
reference:
http://activemq.apache.org/apollo/versions/1.7/website/documentation/user-manual.html
-->
<broker xmlns="http://activemq.apache.org/schema/activemq/apollo">
<notes>
The default configuration with tls/ssl enabled.
</notes>
<log_category console="console" security="security" connection="connection" audit="audit"/>
<authentication domain="apollo"/>
<!-- Give admins full access -->
<access_rule allow="admins" action="*"/>
<access_rule allow="*" action="connect" kind="connector"/>
<virtual_host id="subscriptions">
<!--
You should add all the host names that this virtual host is known as
to properly support the STOMP 1.1 virtual host feature.
-->
<host_name>subscriptions</host_name>
<host_name>localhost</host_name>
<host_name>127.0.0.1</host_name>
<!-- shotgunning the following out of ansible: -->
<host_name>broker0.monsoontools.com</host_name>
<host_name>broker0.internal.monsoontools.com</host_name>
<host_name>broker-store.monsoontools.com</host_name>
<host_name>broker-sub.monsoontools.com</host_name>
<queue id="distribute" tail_buffer="250M" full_policy="drop head" />
<queue id="store" tail_buffer="700M" full_policy="drop head" />
<queue id="cust*" tail_buffer="50M" full_policy="drop head" />
<queue id="dead*" tail_buffer="50M" full_policy="drop head" />
<queue id="brokertorture" tail_buffer="3G" full_policy="drop head" />
<queue id="another" tail_buffer="50M" full_policy="drop head" />
<queue id="fourth" tail_buffer="700M" quota="3G" full_policy="drop head" />
<!-- Uncomment to disable security for the virtual host -->
<!-- <authentication enabled="false"/> -->
<!-- Uncomment to disable security for the virtual host -->
<!-- <authentication enabled="false"/> -->
<access_rule allow="users" action="connect create destroy send receive consume"/>
<!-- You can delete this element if you want to disable persistence for this virtual host -->
<leveldb_store directory="/media/ephemeral0/data"/>
</virtual_host>
<web_admin bind="http://127.0.0.1:61680"/>
<web_admin bind="https://0.0.0.0:61681"/>
<connector id="tcp" bind="tcp://0.0.0.0:61613" connection_limit="2000"/>
<connector id="ssl" bind="tls://0.0.0.0:61614" connection_limit="2000"/>
<connector id="ws" bind="ws://0.0.0.0:61623" connection_limit="2000"/>
<connector id="wss" bind="wss://0.0.0.0:61624" connection_limit="2000"/>
<key_storage file="${apollo.base}/etc/keystore" password="password" key_password="password"/>
</broker>
{code}
> regular, reproducable OutOfMemory exceptions
> --------------------------------------------
>
> Key: APLO-360
> URL: https://issues.apache.org/jira/browse/APLO-360
> Project: ActiveMQ Apollo
> Issue Type: Bug
> Components: apollo-broker
> Affects Versions: 1.7
> Environment: * OS: Amazon Linux AMI release 2014.03 (centos-like)
> * CPU: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GH
> * MemTotal: 3843028 kB
> * Startup command: apollo -ea -server -XX:+AggressiveOpts -XX:+OptimizeStringConcat -XX:+UseFastAccessorMethods -XX:+UseBiasedLocking -Xmx2662m -Xms128m -Dcom.sun.management.jmxremote -Dapollo.home=/opt/apache-apollo-1.7 -Dapollo.base=/var/lib/subscriptions -classpath /opt/apache-apollo-1.7/lib/apollo-boot.jar org.apache.activemq.apollo.boot.Apollo /var/lib/subscriptions/lib\;/opt/apache-apollo-1.7/lib org.apache.activemq.apollo.cli.Apollo run
> * Java: OpenJDK 64-Bit Server VM 1.7.0_55
> Reporter: Daniel Miles
> Attachments: loadtest.tar.bz2, log.tar.bz2
>
>
> Apollo has been failing under load, and I can consistently reproduce it with this formula. I've attached two things to this bug report:
> * All apollo logs
> * The load-test application (both source code and compiled binary)
> After a few hours of load testing (one producer, five consumers, roughly 7000 messages/second, message size 1.4k), leveldb has a "Short Write" exception then Apollo goes into recovery mode and runs out of memory. All logs are attached, relevant section of stacktrace.log looks like this:
> {code}
> 2014-06-25 23:54:55,593 | WARN | stackref=146d56da433
> java.io.IOException: Short write
> at org.apache.activemq.apollo.broker.store.leveldb.RecordLog$LogAppender.flush(RecordLog.scala:206)
> at org.apache.activemq.apollo.broker.store.leveldb.RecordLog.appender(RecordLog.scala:464)
> at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$store$1.apply$mcV$sp(LevelDBClient.scala:870)
> at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:870)
> at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:870)
> at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.using_index(LevelDBClient.scala:664)
> at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$retry_using_index$1.apply(LevelDBClient.scala:670)
> at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.retry(LevelDBClient.scala:758)
> at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.retry_using_index(LevelDBClient.scala:670)
> at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.store(LevelDBClient.scala:869)
> at org.apache.activemq.apollo.broker.store.leveldb.LevelDBStore$$anonfun$store$1.apply$mcV$sp(LevelDBStore.scala:82)
> at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-06-26 00:10:39,219 | WARN | stackref=146d56da434
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:14:44,191 | WARN | stackref=146d56da435
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:16:40,941 | WARN | stackref=146d56da438
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:22:30,467 | WARN | stackref=146d56da439
> java.lang.OutOfMemoryError: Java heap space
> {code}
> And the same section of apollo.log looks like this:
> {code}
> 2014-06-25 23:54:55,584 | WARN | DB operation failed. (entering recovery mode): java.io.IOException: Short write | 146d56da433
> 2014-06-26 00:01:35,273 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> at java.io.ByteArrayOutputStream.<init>(ByteArrayOutputStream.java:77)
> at sun.security.ssl.OutputRecord.<init>(OutputRecord.java:76)
> at sun.security.ssl.EngineOutputRecord.<init>(EngineOutputRecord.java:65)
> at sun.security.ssl.HandshakeOutStream.<init>(HandshakeOutStream.java:63)
> at sun.security.ssl.Handshaker.activate(Handshaker.java:491)
> at sun.security.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:711)
> at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:809)
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:761)
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
> at org.eclipse.jetty.io.nio.SslConnection.unwrap(SslConnection.java:519)
> at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:354)
> at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:43)
> at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.fill(SslConnection.java:661)
> at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1030)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:275)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
> at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622)
> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-06-26 00:04:00,378 | WARN | |
> java.lang.OutOfMemoryError: Java heap space
> at sun.security.ssl.InputRecord.<init>(InputRecord.java:84)
> at sun.security.ssl.EngineInputRecord.<init>(EngineInputRecord.java:63)
> at sun.security.ssl.SSLEngineImpl.init(SSLEngineImpl.java:407)
> at sun.security.ssl.SSLEngineImpl.<init>(SSLEngineImpl.java:343)
> at sun.security.ssl.SSLContextImpl.engineCreateSSLEngine(SSLContextImpl.java:208)
> at javax.net.ssl.SSLContext.createSSLEngine(SSLContext.java:362)
> at org.eclipse.jetty.util.ssl.SslContextFactory.newSslEngine(SslContextFactory.java:1495)
> at org.eclipse.jetty.server.ssl.SslSelectChannelConnector.createSSLEngine(SslSelectChannelConnector.java:588)
> at org.eclipse.jetty.server.ssl.SslSelectChannelConnector.newConnection(SslSelectChannelConnector.java:551)
> at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.newConnection(SelectChannelConnector.java:320)
> at org.eclipse.jetty.server.nio.SelectChannelConnector.newEndPoint(SelectChannelConnector.java:267)
> at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.newEndPoint(SelectChannelConnector.java:326)
> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.createEndPoint(SelectorManager.java:836)
> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:491)
> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-06-26 00:10:39,219 | WARN | java.lang.OutOfMemoryError: Java heap space | 146d56da434
> 2014-06-26 00:10:39,220 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> at java.math.BigInteger.multiplyToLen(BigInteger.java:1226)
> at java.math.BigInteger.oddModPow(BigInteger.java:1764)
> at java.math.BigInteger.modPow(BigInteger.java:1599)
> at sun.security.rsa.RSACore.crtCrypt(RSACore.java:154)
> at sun.security.rsa.RSACore.rsa(RSACore.java:101)
> at sun.security.rsa.RSASignature.engineSign(RSASignature.java:177)
> at java.security.Signature$Delegate.engineSign(Signature.java:1160)
> at java.security.Signature.sign(Signature.java:553)
> at sun.security.ssl.HandshakeMessage$ECDH_ServerKeyExchange.<init>(HandshakeMessage.java:1012)
> at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:772)
> at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:167)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:808)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:806)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1227)
> at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:370)
> at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:43)
> at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.fill(SslConnection.java:661)
> at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1030)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:275)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
> at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622)
> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-06-26 00:12:09,137 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:12:54,045 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:14:44,191 | WARN | java.lang.OutOfMemoryError: Java heap space | 146d56da435
> 2014-06-26 00:16:33,678 | WARN | java.lang.OutOfMemoryError: Java heap space | 146d56da438
> 2014-06-26 00:20:41,583 | WARN | |
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:21:53,264 | WARN | |
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:22:07,730 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:22:07,731 | WARN | java.lang.OutOfMemoryError: Java heap space | 146d56da439
> 2014-06-26 00:22:07,731 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:23:28,222 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:23:49,662 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:25:22,976 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:25:37,279 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:26:05,740 | WARN | |
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:26:34,485 | WARN | |
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:27:17,858 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:27:39,331 | WARN | handle failed |
> java.lang.OutOfMemoryError: Java heap space
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)