You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Cott Lang (JIRA)" <ji...@apache.org> on 2016/10/03 17:09:20 UTC
[jira] [Created] (AMQ-6451) Preallocation strategy of zeros can
leak file descriptors causing crash
Cott Lang created AMQ-6451:
------------------------------
Summary: Preallocation strategy of zeros can leak file descriptors causing crash
Key: AMQ-6451
URL: https://issues.apache.org/jira/browse/AMQ-6451
Project: ActiveMQ
Issue Type: Bug
Components: KahaDB
Affects Versions: 5.14.0
Reporter: Cott Lang
Priority: Critical
If the preallocationStrategy is set to 'zeros', ActiveMQ can intermittently become unable to allocate direct buffer memory with the default JVM settings. The exception isn't handled, and ends up both creating an empty journal file and, more importantly, leaking a file descriptor.
ActiveMQ eventually runs out of file descriptors and crashes.
In addition to handling this condition, perhaps the default ACTIVEMQ_OPTS_MEMORY settings should configure enough direct memory to allow some multiple of log files to be created near simultaneously, or at least this possibility documented in the KahaDB settings.
Relevant logs:
{noformat}
2016-10-03 12:47:24,154 | WARN | Async error occurred: java.lang.OutOfMemoryError: Direct buffer memory | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport: ssl:///x.x.x.x:60805
2016-10-03 12:47:24,818 | WARN | Async error occurred: java.lang.OutOfMemoryError: Direct buffer memory | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport: ssl:///x.x.x.x:60811
2016-10-03 12:47:25,477 | WARN | Async error occurred: java.lang.OutOfMemoryError: Direct buffer memory | org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport: ssl:///x.x.x.x:49830
2016-10-03 12:47:26,146 | WARN | Store COMMIT FAILED: | org.apache.activemq.transaction.LocalTransaction | ActiveMQ Transport: ssl:///x.x.x.x:65534
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:693)[:1.8.0_101]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)[:1.8.0_101]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)[:1.8.0_101]
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)[:1.8.0_101]
at sun.nio.ch.IOUtil.write(IOUtil.java:58)[:1.8.0_101]
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)[:1.8.0_101]
at org.apache.activemq.store.kahadb.disk.journal.Journal.doPreallocationZeros(Journal.java:366)[activemq-kahadb-store-5.14.0.jar:5.14.0]
at org.apache.activemq.store.kahadb.disk.journal.Journal.preallocateEntireJournalDataFile(Journal.java:333)[activemq-kahadb-store-5.14.0.jar:5.14.0]
at org.apache.activemq.store.kahadb.disk.journal.Journal.newDataFile(Journal.java:631)[activemq-kahadb-store-5.14.0.jar:5.14.0]
at org.apache.activemq.store.kahadb.disk.journal.Journal.rotateWriteFile(Journal.java:595)[activemq-kahadb-store-5.14.0.jar:5.14.0]
at org.apache.activemq.store.kahadb.disk.journal.Journal.getCurrentDataFile(Journal.java:984)[activemq-kahadb-store-5.14.0.jar:5.14.0]
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.enqueue(DataFileAppender.java:189)[activemq-kahadb-store-5.14.0.jar:5.14.0]
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.storeItem(DataFileAppender.java:128)[activemq-kahadb-store-5.14.0.jar:5.14.0]
at org.apache.activemq.store.kahadb.disk.journal.Journal.write(Journal.java:890)[activemq-kahadb-store-5.14.0.jar:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1108)[activemq-kahadb-store-5.14.0.jar:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1090)[activemq-kahadb-store-5.14.0.jar:5.14.0]
at org.apache.activemq.store.kahadb.KahaDBTransactionStore.commit(KahaDBTransactionStore.java:301)[activemq-kahadb-store-5.14.0.jar:5.14.0]
at org.apache.activemq.transaction.LocalTransaction.commit(LocalTransaction.java:70)[activemq-broker-5.14.0.jar:5.14.0]
at org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:252)[activemq-broker-5.14.0.jar:5.14.0]
at org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:113)[activemq-broker-5.14.0.jar:5.14.0]
at org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:113)[activemq-broker-5.14.0.jar:5.14.0]
at org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:118)[activemq-broker-5.14.0.jar:5.14.0]
at org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:535)[activemq-broker-5.14.0.jar:5.14.0]
at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)[activemq-client-5.14.0.jar:5.14.0]
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:333)[activemq-broker-5.14.0.jar:5.14.0]
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:197)[activemq-broker-5.14.0.jar:5.14.0]
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.14.0.jar:5.14.0]
at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)[activemq-client-5.14.0.jar:5.14.0]
at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)[activemq-client-5.14.0.jar:5.14.0]
at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.14.0.jar:5.14.0]
at org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:108)[activemq-client-5.14.0.jar:5.14.0]
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.14.0.jar:5.14.0]
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.14.0.jar:5.14.0]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_101]
{noformat}
Empty journal files:
{noformat}
-rw-r--r-- 1 activemq activemq 0 Oct 3 12:47 db-51425.log
-rw-r--r-- 1 activemq activemq 0 Oct 3 12:47 db-51426.log
-rw-r--r-- 1 activemq activemq 0 Oct 3 12:47 db-51427.log
-rw-r--r-- 1 activemq activemq 0 Oct 3 12:47 db-51428.log
-rw-r--r-- 1 activemq activemq 0 Oct 3 12:47 db-51429.log
{noformat}
lsof output:
{noformat}
java 29263 activemq 452u REG 202,48 0 160 /activemq/kahadb/db-51425.log
java 29263 activemq 455u REG 202,48 0 153 /activemq/kahadb/db-51426.log
java 29263 activemq 456u REG 202,48 0 133 /activemq/kahadb/db-51427.log
java 29263 activemq 462u REG 202,48 0 157 /activemq/kahadb/db-51428.log
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)