You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Cybexion <cy...@email.de> on 2009/01/29 09:37:56 UTC
OutOfMemoryError after 9505925 persistent messages
Hi,
I'm running apache-activemq-5.2.0.1-fuse on jdk 1.6.
When I was doing a high load 24/7 test where I produced 9505925 persistent
small message (kaha) suddenly amq stopped working with an
OutOfMemoryException.
My memory settings are
-Xmn100M \
-Xms500M \
-Xmx1024M
The Exception looks like this:
Exception in thread "ActiveMQ Connection Dispatcher: /127.0.0.1:32929"
java.lang.OutOfMemoryError: Java heap space
at java.util.jar.Attributes.read(Attributes.java:377)
at java.util.jar.Manifest.read(Manifest.java:182)
at java.util.jar.Manifest.<init>(Manifest.java:52)
at java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
at java.util.jar.JarFile.getManifest(JarFile.java:146)
at
sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at
org.apache.activemq.broker.TransportConnection.serviceExceptionAsync(TransportConnection.java:244)
at
org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:841)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Exception in thread "ActiveMQ Scheduler" java.lang.OutOfMemoryError: Java
heap space
at java.util.HashMap.addEntry(HashMap.java:753)
at java.util.HashMap.put(HashMap.java:385)
at java.util.HashSet.add(HashSet.java:200)
at java.util.AbstractCollection.addAll(AbstractCollection.java:305)
at
org.apache.activemq.store.amq.AMQPersistenceAdapter.cleanup(AMQPersistenceAdapter.java:439)
at
org.apache.activemq.store.amq.AMQPersistenceAdapter$3.run(AMQPersistenceAdapter.java:279)
at
org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError:
Java heap space
at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:679)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
Exception in thread "RMI TCP Connection(idle)" at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError:
Java heap space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError:
Java heap space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError:
Java heap space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError:
Java heap space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError:
Java heap space
Exception in thread "InactivityMonitor WriteCheck"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError:
Java heap space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError:
Java heap space
...
The stdout looks like this:
2009-01-28 10:48:35,664 [ueue://tracking] DEBUG
vemq.store.amq.AMQMessageStore - Batch update done.
2009-01-28 10:48:33,537 [iveMQ Scheduler] DEBUG
tore.amq.AMQPersistenceAdapter - lastDataFile: 5663
2009-01-28 10:48:27,177 [itor WriteCheck] DEBUG
.transport.InactivityMonitor$2 - 72107 ms elapsed since last write check.
2009-01-28 10:48:56,877 [iveMQ Scheduler] DEBUG
ha.impl.async.AsyncDataManager - lastFileId=5662, purgeList: (0) []
2009-01-28 10:48:52,634 [ueue://tracking] DEBUG
vemq.store.amq.AMQMessageStore - Doing batch update... adding: 3 removing: 4
2009-01-28 10:48:48,395 [127.0.0.1:39844] DEBUG
vemq.store.amq.AMQMessageStore - Journalled message add for:
ID:adbl006-40098-1233049753494-0:38:3395:1:2,
at: offset = 168080, file = 5663, size = 384, type = 1
2009-01-28 10:48:48,395 [127.0.0.1:32929] DEBUG
vemq.store.amq.AMQMessageStore - Journalled message remove for:
ID:adbl006-40098-1233049753494-0:30:278402
8:1:1, at: offset = 167801, file = 5663, size = 279, type = 1
2009-01-28 10:48:37,784 [nitor ReadCheck] DEBUG
.transport.InactivityMonitor$1 - 27573 ms elapsed since last read check.
2009-01-28 10:51:32,122 [127.0.0.1:39844] DEBUG
vemq.store.amq.AMQMessageStore - Journalled message add for:
ID:adbl006-40098-1233049753494-0:38:3392:1:2,
at: offset = 168464, file = 5663, size = 384, type = 1
2009-01-28 10:51:27,528 [ueue://tracking] DEBUG
vemq.store.amq.AMQMessageStore - Batch update done.
2009-01-28 10:50:48,614 [itor WriteCheck] DEBUG
.transport.InactivityMonitor$2 - 165814 ms elapsed since last write check.
2009-01-28 10:50:48,614 [iveMQ Scheduler] DEBUG
tore.amq.AMQPersistenceAdapter - dataFilesInProgress.values: (1) [{5663=3}]
2009-01-28 10:51:40,604 [ueue://tracking] DEBUG
vemq.store.amq.AMQMessageStore - Doing batch update... adding: 2 removing: 1
2009-01-28 10:51:36,365 [127.0.0.1:39844] DEBUG
vemq.store.amq.AMQMessageStore - Journalled message add for:
ID:adbl006-40098-1233049753494-0:38:3391:1:2,
at: offset = 169127, file = 5663, size = 384, type = 1
2009-01-28 10:51:32,122 [127.0.0.1:32929] DEBUG
vemq.store.amq.AMQMessageStore - Journalled message remove for:
ID:adbl006-40098-1233049753494-0:30:278403
2:1:1, at: offset = 168848, file = 5663, size = 279, type = 1
2009-01-28 10:51:32,122 [nitor ReadCheck] DEBUG
.transport.InactivityMonitor$1 - 197674 ms elapsed since last read check.
2009-01-28 10:55:28,803 [127.0.0.1:39844] DEBUG
vemq.store.amq.AMQMessageStore - Journalled message add for:
ID:adbl006-40098-1233049753494-0:38:3385:1:2,
at: offset = 169511, file = 5663, size = 384, type = 1
2009-01-28 10:55:28,803 [ueue://tracking] DEBUG
vemq.store.amq.AMQMessageStore - Batch update done.
2009-01-28 11:09:10,234 [.21.150.21:8161] ERROR
rg.slf4j.impl.JCLLoggerAdapter - EXCEPTION
java.lang.OutOfMemoryError: Java heap space
2009-01-28 10:58:33,322 [127.0.0.1:39844] DEBUG
vemq.store.amq.AMQMessageStore - Journalled message add for:
ID:adbl006-40098-1233049753494-0:38:3383:1:2,
at: offset = 170174, file = 5663, size = 384, type = 1
2009-01-28 10:55:51,594 [127.0.0.1:32929] DEBUG
vemq.store.amq.AMQMessageStore - Journalled message remove for:
ID:adbl006-40098-1233049753494-0:30:278372
5:1:7, at: offset = 169895, file = 5663, size = 279, type = 1
2009-01-28 10:55:42,624 [nitor ReadCheck] DEBUG
.transport.InactivityMonitor$1 - 424840 ms elapsed since last read check.
2009-01-28 11:17:46,299 [127.0.0.1:39844] DEBUG
vemq.store.amq.AMQMessageStore - Journalled message add for:
ID:adbl006-40098-1233049753494-0:38:2915:1:4,
at: offset = 170558, file = 5663, size = 384, type = 1
2009-01-28 11:17:03,736 [127.0.0.1:32929] DEBUG
vemq.store.amq.AMQMessageStore - Journalled message remove for:
ID:adbl006-40098-1233049753494-0:30:278403
3:1:1, at: offset = 170942, file = 5663, size = 279, type = 1
2009-01-28 11:13:41,738 [3895909@qtp0-14] ERROR
rg.slf4j.impl.JCLLoggerAdapter - /admin/queues.jsp
java.lang.OutOfMemoryError: Java heap space
2009-01-28 11:09:25,197 [Timer-0 ] ERROR
rg.slf4j.impl.JCLLoggerAdapter - Problem scavenging sessions
java.lang.OutOfMemoryError: Java heap space
2009-01-28 11:19:30,007 [127.0.0.1:32929] DEBUG
vemq.store.amq.AMQMessageStore - Journalled message remove for:
ID:adbl006-40098-1233049753494-0:30:278403
5:1:1, at: offset = 171605, file = 5663, size = 279, type = 1
2009-01-28 11:18:50,504 [127.0.0.1:39844] DEBUG
vemq.store.amq.AMQMessageStore - Journalled message add for:
ID:adbl006-40098-1233049753494-0:38:3328:1:3,
at: offset = 171221, file = 5663, size = 384, type = 1
2009-01-28 11:17:46,299 [nitor ReadCheck] DEBUG
.transport.InactivityMonitor$1 - 1574177 ms elapsed since last read check.
2009-01-28 11:30:42,525 [127.0.0.1:32929] DEBUG
emq.broker.TransportConnection - Transport failed: java.io.IOException:
Unexpected error occured
java.io.IOException: Unexpected error occured
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:192)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap space
2009-01-28 11:31:20,726 [127.0.0.1:39844] DEBUG
emq.broker.TransportConnection - Transport failed: java.io.IOException:
Unexpected error occured
java.io.IOException: Unexpected error occured
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:192)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap space
2009-01-28 11:39:58,676 [127.0.0.1:39844] DEBUG
emq.broker.TransportConnection - Stopping connection: /127.0.0.1:39844
2009-01-28 11:40:07,371 [127.0.0.1:32929] DEBUG
emq.broker.TransportConnection - Stopping connection: /127.0.0.1:32929
2009-01-28 11:41:04,406 [127.0.0.1:39844] DEBUG
emq.transport.tcp.TcpTransport - Stopping transport tcp:///127.0.0.1:39844
2009-01-28 11:41:26,226 [127.0.0.1:32929] DEBUG
emq.transport.tcp.TcpTransport - Stopping transport tcp:///127.0.0.1:32929
2009-01-28 11:43:11,185 [127.0.0.1:39844] DEBUG
emq.broker.TransportConnection - Stopped transport: /127.0.0.1:39844
2009-01-28 11:47:36,654 [127.0.0.1:32929] DEBUG
emq.broker.TransportConnection - Stopped transport: /127.0.0.1:32929
2009-01-28 11:52:06,738 [127.0.0.1:32929] DEBUG
emq.broker.TransportConnection - Cleaning up connection resources:
/127.0.0.1:32929
2009-01-28 11:53:30,458 [127.0.0.1:32929] DEBUG
q.broker.region.AbstractRegion - Removing consumer:
ID:adbl006-40098-1233049753494-0:0:-1:1
2009-01-28 11:55:40,444 [127.0.0.1:32929] DEBUG
q.broker.region.AbstractRegion - Removing consumer:
ID:adbl006-40098-1233049753494-0:0:1:1
2009-01-28 11:51:49,128 [127.0.0.1:39844] DEBUG
emq.broker.TransportConnection - Cleaning up connection resources:
/127.0.0.1:39844
2009-01-28 11:56:07,668 [127.0.0.1:32929] WARN
emq.broker.TransportConnection - Failed to remove consumer:
ID:adbl006-40098-1233049753494-0:0:1:1. Reason
: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
2009-01-28 11:56:23,041 [127.0.0.1:32929] DEBUG
emq.broker.TransportConnection - Connection Stopped: /127.0.0.1:32929
2009-01-28 11:56:23,040 [127.0.0.1:39844] DEBUG
q.broker.region.AbstractRegion - Removing consumer:
ID:adbl006-40098-1233049753494-0:38:-1:1
2009-01-28 11:56:46,289 [127.0.0.1:39844] DEBUG
emq.broker.TransportConnection - Connection Stopped: /127.0.0.1:39844
Did anyone else detect this behavoir?
Best regards
Jörg
--
View this message in context: http://www.nabble.com/OutOfMemoryError-after-9505925-persistent-messages-tp21722796p21722796.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OutOfMemoryError after 9505925 persistent messages
Posted by Cybexion <cy...@email.de>.
sorry, have no junit test.
But I created an Issue with all information inside:
https://issues.apache.org/activemq/browse/AMQ-2090
rajdavies wrote:
>
>
> On 29 Jan 2009, at 08:37, Cybexion wrote:
>
>>
>> Hi,
>>
>> I'm running apache-activemq-5.2.0.1-fuse on jdk 1.6.
>> When I was doing a high load 24/7 test where I produced 9505925
>> persistent
>> small message (kaha) suddenly amq stopped working with an
>> OutOfMemoryException.
>> My memory settings are
>> -Xmn100M \
>> -Xms500M \
>> -Xmx1024M
>>
>> The Exception looks like this:
>>
>> Exception in thread "ActiveMQ Connection Dispatcher: /127.0.0.1:32929"
>> java.lang.OutOfMemoryError: Java heap space
>> at java.util.jar.Attributes.read(Attributes.java:377)
>> at java.util.jar.Manifest.read(Manifest.java:182)
>> at java.util.jar.Manifest.<init>(Manifest.java:52)
>> at
>> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>> at java.util.jar.JarFile.getManifest(JarFile.java:146)
>> at
>> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
>> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
>> 320)
>> at
>> org
>> .apache
>> .activemq
>> .broker
>> .TransportConnection.serviceExceptionAsync(TransportConnection.java:
>> 244)
>> at
>> org
>> .apache
>> .activemq
>> .broker.TransportConnection.iterate(TransportConnection.java:841)
>> at
>> org
>> .apache
>> .activemq
>> .thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
>> at
>> org.apache.activemq.thread.DedicatedTaskRunner
>> $1.run(DedicatedTaskRunner.java:36)
>> Exception in thread "ActiveMQ Scheduler" java.lang.OutOfMemoryError:
>> Java
>> heap space
>> at java.util.HashMap.addEntry(HashMap.java:753)
>> at java.util.HashMap.put(HashMap.java:385)
>> at java.util.HashSet.add(HashSet.java:200)
>> at
>> java.util.AbstractCollection.addAll(AbstractCollection.java:305)
>> at
>> org
>> .apache
>> .activemq
>> .store.amq.AMQPersistenceAdapter.cleanup(AMQPersistenceAdapter.java:
>> 439)
>> at
>> org.apache.activemq.store.amq.AMQPersistenceAdapter
>> $3.run(AMQPersistenceAdapter.java:279)
>> at
>> org
>> .apache
>> .activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>> at java.util.TimerThread.mainLoop(Timer.java:512)
>> at java.util.TimerThread.run(Timer.java:462)
>> Exception in thread "RMI TCP Connection(idle)"
>> java.lang.OutOfMemoryError:
>> Java heap space
>> at
>> java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
>> at
>> java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
>> at
>> sun.rmi.transport.tcp.TCPTransport
>> $ConnectionHandler.run0(TCPTransport.java:679)
>> at
>> sun.rmi.transport.tcp.TCPTransport
>> $ConnectionHandler.run(TCPTransport.java:649)
>> Exception in thread "RMI TCP Connection(idle)" at
>> java.util.concurrent.ThreadPoolExecutor
>> $Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor
>> $Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> java.lang.OutOfMemoryError: Java heap space
>> Exception in thread "RMI TCP Connection(idle)"
>> java.lang.OutOfMemoryError:
>> Java heap space
>> Exception in thread "RMI TCP Connection(idle)"
>> java.lang.OutOfMemoryError:
>> Java heap space
>> Exception in thread "RMI TCP Connection(idle)"
>> java.lang.OutOfMemoryError:
>> Java heap space
>> Exception in thread "RMI TCP Connection(idle)"
>> java.lang.OutOfMemoryError:
>> Java heap space
>> Exception in thread "RMI TCP Connection(idle)"
>> java.lang.OutOfMemoryError:
>> Java heap space
>> Exception in thread "InactivityMonitor WriteCheck"
>> java.lang.OutOfMemoryError: Java heap space
>> Exception in thread "RMI TCP Connection(idle)"
>> java.lang.OutOfMemoryError:
>> Java heap space
>> Exception in thread "RMI TCP Connection(idle)"
>> java.lang.OutOfMemoryError:
>> Java heap space
>> ...
>>
>>
>> The stdout looks like this:
>> 2009-01-28 10:48:35,664 [ueue://tracking] DEBUG
>> vemq.store.amq.AMQMessageStore - Batch update done.
>> 2009-01-28 10:48:33,537 [iveMQ Scheduler] DEBUG
>> tore.amq.AMQPersistenceAdapter - lastDataFile: 5663
>> 2009-01-28 10:48:27,177 [itor WriteCheck] DEBUG
>> .transport.InactivityMonitor$2 - 72107 ms elapsed since last write
>> check.
>> 2009-01-28 10:48:56,877 [iveMQ Scheduler] DEBUG
>> ha.impl.async.AsyncDataManager - lastFileId=5662, purgeList: (0) []
>> 2009-01-28 10:48:52,634 [ueue://tracking] DEBUG
>> vemq.store.amq.AMQMessageStore - Doing batch update... adding: 3
>> removing: 4
>> 2009-01-28 10:48:48,395 [127.0.0.1:39844] DEBUG
>> vemq.store.amq.AMQMessageStore - Journalled message add for:
>> ID:adbl006-40098-1233049753494-0:38:3395:1:2,
>> at: offset = 168080, file = 5663, size = 384, type = 1
>> 2009-01-28 10:48:48,395 [127.0.0.1:32929] DEBUG
>> vemq.store.amq.AMQMessageStore - Journalled message remove for:
>> ID:adbl006-40098-1233049753494-0:30:278402
>> 8:1:1, at: offset = 167801, file = 5663, size = 279, type = 1
>> 2009-01-28 10:48:37,784 [nitor ReadCheck] DEBUG
>> .transport.InactivityMonitor$1 - 27573 ms elapsed since last read
>> check.
>> 2009-01-28 10:51:32,122 [127.0.0.1:39844] DEBUG
>> vemq.store.amq.AMQMessageStore - Journalled message add for:
>> ID:adbl006-40098-1233049753494-0:38:3392:1:2,
>> at: offset = 168464, file = 5663, size = 384, type = 1
>> 2009-01-28 10:51:27,528 [ueue://tracking] DEBUG
>> vemq.store.amq.AMQMessageStore - Batch update done.
>> 2009-01-28 10:50:48,614 [itor WriteCheck] DEBUG
>> .transport.InactivityMonitor$2 - 165814 ms elapsed since last write
>> check.
>> 2009-01-28 10:50:48,614 [iveMQ Scheduler] DEBUG
>> tore.amq.AMQPersistenceAdapter - dataFilesInProgress.values: (1)
>> [{5663=3}]
>> 2009-01-28 10:51:40,604 [ueue://tracking] DEBUG
>> vemq.store.amq.AMQMessageStore - Doing batch update... adding: 2
>> removing: 1
>> 2009-01-28 10:51:36,365 [127.0.0.1:39844] DEBUG
>> vemq.store.amq.AMQMessageStore - Journalled message add for:
>> ID:adbl006-40098-1233049753494-0:38:3391:1:2,
>> at: offset = 169127, file = 5663, size = 384, type = 1
>> 2009-01-28 10:51:32,122 [127.0.0.1:32929] DEBUG
>> vemq.store.amq.AMQMessageStore - Journalled message remove for:
>> ID:adbl006-40098-1233049753494-0:30:278403
>> 2:1:1, at: offset = 168848, file = 5663, size = 279, type = 1
>> 2009-01-28 10:51:32,122 [nitor ReadCheck] DEBUG
>> .transport.InactivityMonitor$1 - 197674 ms elapsed since last read
>> check.
>> 2009-01-28 10:55:28,803 [127.0.0.1:39844] DEBUG
>> vemq.store.amq.AMQMessageStore - Journalled message add for:
>> ID:adbl006-40098-1233049753494-0:38:3385:1:2,
>> at: offset = 169511, file = 5663, size = 384, type = 1
>> 2009-01-28 10:55:28,803 [ueue://tracking] DEBUG
>> vemq.store.amq.AMQMessageStore - Batch update done.
>> 2009-01-28 11:09:10,234 [.21.150.21:8161] ERROR
>> rg.slf4j.impl.JCLLoggerAdapter - EXCEPTION
>> java.lang.OutOfMemoryError: Java heap space
>> 2009-01-28 10:58:33,322 [127.0.0.1:39844] DEBUG
>> vemq.store.amq.AMQMessageStore - Journalled message add for:
>> ID:adbl006-40098-1233049753494-0:38:3383:1:2,
>> at: offset = 170174, file = 5663, size = 384, type = 1
>> 2009-01-28 10:55:51,594 [127.0.0.1:32929] DEBUG
>> vemq.store.amq.AMQMessageStore - Journalled message remove for:
>> ID:adbl006-40098-1233049753494-0:30:278372
>> 5:1:7, at: offset = 169895, file = 5663, size = 279, type = 1
>> 2009-01-28 10:55:42,624 [nitor ReadCheck] DEBUG
>> .transport.InactivityMonitor$1 - 424840 ms elapsed since last read
>> check.
>> 2009-01-28 11:17:46,299 [127.0.0.1:39844] DEBUG
>> vemq.store.amq.AMQMessageStore - Journalled message add for:
>> ID:adbl006-40098-1233049753494-0:38:2915:1:4,
>> at: offset = 170558, file = 5663, size = 384, type = 1
>> 2009-01-28 11:17:03,736 [127.0.0.1:32929] DEBUG
>> vemq.store.amq.AMQMessageStore - Journalled message remove for:
>> ID:adbl006-40098-1233049753494-0:30:278403
>> 3:1:1, at: offset = 170942, file = 5663, size = 279, type = 1
>> 2009-01-28 11:13:41,738 [3895909@qtp0-14] ERROR
>> rg.slf4j.impl.JCLLoggerAdapter - /admin/queues.jsp
>> java.lang.OutOfMemoryError: Java heap space
>> 2009-01-28 11:09:25,197 [Timer-0 ] ERROR
>> rg.slf4j.impl.JCLLoggerAdapter - Problem scavenging sessions
>> java.lang.OutOfMemoryError: Java heap space
>> 2009-01-28 11:19:30,007 [127.0.0.1:32929] DEBUG
>> vemq.store.amq.AMQMessageStore - Journalled message remove for:
>> ID:adbl006-40098-1233049753494-0:30:278403
>> 5:1:1, at: offset = 171605, file = 5663, size = 279, type = 1
>> 2009-01-28 11:18:50,504 [127.0.0.1:39844] DEBUG
>> vemq.store.amq.AMQMessageStore - Journalled message add for:
>> ID:adbl006-40098-1233049753494-0:38:3328:1:3,
>> at: offset = 171221, file = 5663, size = 384, type = 1
>> 2009-01-28 11:17:46,299 [nitor ReadCheck] DEBUG
>> .transport.InactivityMonitor$1 - 1574177 ms elapsed since last read
>> check.
>> 2009-01-28 11:30:42,525 [127.0.0.1:32929] DEBUG
>> emq.broker.TransportConnection - Transport failed:
>> java.io.IOException:
>> Unexpected error occured
>> java.io.IOException: Unexpected error occured
>> at
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>> 192)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>> 2009-01-28 11:31:20,726 [127.0.0.1:39844] DEBUG
>> emq.broker.TransportConnection - Transport failed:
>> java.io.IOException:
>> Unexpected error occured
>> java.io.IOException: Unexpected error occured
>> at
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>> 192)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>> 2009-01-28 11:39:58,676 [127.0.0.1:39844] DEBUG
>> emq.broker.TransportConnection - Stopping connection: /127.0.0.1:39844
>> 2009-01-28 11:40:07,371 [127.0.0.1:32929] DEBUG
>> emq.broker.TransportConnection - Stopping connection: /127.0.0.1:32929
>> 2009-01-28 11:41:04,406 [127.0.0.1:39844] DEBUG
>> emq.transport.tcp.TcpTransport - Stopping transport tcp:///
>> 127.0.0.1:39844
>> 2009-01-28 11:41:26,226 [127.0.0.1:32929] DEBUG
>> emq.transport.tcp.TcpTransport - Stopping transport tcp:///
>> 127.0.0.1:32929
>> 2009-01-28 11:43:11,185 [127.0.0.1:39844] DEBUG
>> emq.broker.TransportConnection - Stopped transport: /127.0.0.1:39844
>> 2009-01-28 11:47:36,654 [127.0.0.1:32929] DEBUG
>> emq.broker.TransportConnection - Stopped transport: /127.0.0.1:32929
>> 2009-01-28 11:52:06,738 [127.0.0.1:32929] DEBUG
>> emq.broker.TransportConnection - Cleaning up connection resources:
>> /127.0.0.1:32929
>> 2009-01-28 11:53:30,458 [127.0.0.1:32929] DEBUG
>> q.broker.region.AbstractRegion - Removing consumer:
>> ID:adbl006-40098-1233049753494-0:0:-1:1
>> 2009-01-28 11:55:40,444 [127.0.0.1:32929] DEBUG
>> q.broker.region.AbstractRegion - Removing consumer:
>> ID:adbl006-40098-1233049753494-0:0:1:1
>> 2009-01-28 11:51:49,128 [127.0.0.1:39844] DEBUG
>> emq.broker.TransportConnection - Cleaning up connection resources:
>> /127.0.0.1:39844
>> 2009-01-28 11:56:07,668 [127.0.0.1:32929] WARN
>> emq.broker.TransportConnection - Failed to remove consumer:
>> ID:adbl006-40098-1233049753494-0:0:1:1. Reason
>> : java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> 2009-01-28 11:56:23,041 [127.0.0.1:32929] DEBUG
>> emq.broker.TransportConnection - Connection Stopped: /127.0.0.1:32929
>> 2009-01-28 11:56:23,040 [127.0.0.1:39844] DEBUG
>> q.broker.region.AbstractRegion - Removing consumer:
>> ID:adbl006-40098-1233049753494-0:38:-1:1
>> 2009-01-28 11:56:46,289 [127.0.0.1:39844] DEBUG
>> emq.broker.TransportConnection - Connection Stopped: /127.0.0.1:39844
>>
>>
>> Did anyone else detect this behavoir?
>>
>> Best regards
>>
>> Jörg
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/OutOfMemoryError-after-9505925-persistent-messages-tp21722796p21722796.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
> Haven't seen that before - could you raise an issue and create a junit
> test case for us ?
>
> cheers,
>
> Rob
>
> Rob Davies
> http://fusesource.com
> http://rajdavies.blogspot.com/
>
>
>
>
>
>
>
>
--
View this message in context: http://www.nabble.com/OutOfMemoryError-after-9505925-persistent-messages-tp21722796p21725287.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OutOfMemoryError after 9505925 persistent messages
Posted by Rob Davies <ra...@gmail.com>.
On 29 Jan 2009, at 08:37, Cybexion wrote:
>
> Hi,
>
> I'm running apache-activemq-5.2.0.1-fuse on jdk 1.6.
> When I was doing a high load 24/7 test where I produced 9505925
> persistent
> small message (kaha) suddenly amq stopped working with an
> OutOfMemoryException.
> My memory settings are
> -Xmn100M \
> -Xms500M \
> -Xmx1024M
>
> The Exception looks like this:
>
> Exception in thread "ActiveMQ Connection Dispatcher: /127.0.0.1:32929"
> java.lang.OutOfMemoryError: Java heap space
> at java.util.jar.Attributes.read(Attributes.java:377)
> at java.util.jar.Manifest.read(Manifest.java:182)
> at java.util.jar.Manifest.<init>(Manifest.java:52)
> at
> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
> at java.util.jar.JarFile.getManifest(JarFile.java:146)
> at
> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
> 320)
> at
> org
> .apache
> .activemq
> .broker
> .TransportConnection.serviceExceptionAsync(TransportConnection.java:
> 244)
> at
> org
> .apache
> .activemq
> .broker.TransportConnection.iterate(TransportConnection.java:841)
> at
> org
> .apache
> .activemq
> .thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> at
> org.apache.activemq.thread.DedicatedTaskRunner
> $1.run(DedicatedTaskRunner.java:36)
> Exception in thread "ActiveMQ Scheduler" java.lang.OutOfMemoryError:
> Java
> heap space
> at java.util.HashMap.addEntry(HashMap.java:753)
> at java.util.HashMap.put(HashMap.java:385)
> at java.util.HashSet.add(HashSet.java:200)
> at
> java.util.AbstractCollection.addAll(AbstractCollection.java:305)
> at
> org
> .apache
> .activemq
> .store.amq.AMQPersistenceAdapter.cleanup(AMQPersistenceAdapter.java:
> 439)
> at
> org.apache.activemq.store.amq.AMQPersistenceAdapter
> $3.run(AMQPersistenceAdapter.java:279)
> at
> org
> .apache
> .activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> Exception in thread "RMI TCP Connection(idle)"
> java.lang.OutOfMemoryError:
> Java heap space
> at
> java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
> at
> java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
> at
> sun.rmi.transport.tcp.TCPTransport
> $ConnectionHandler.run0(TCPTransport.java:679)
> at
> sun.rmi.transport.tcp.TCPTransport
> $ConnectionHandler.run(TCPTransport.java:649)
> Exception in thread "RMI TCP Connection(idle)" at
> java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "RMI TCP Connection(idle)"
> java.lang.OutOfMemoryError:
> Java heap space
> Exception in thread "RMI TCP Connection(idle)"
> java.lang.OutOfMemoryError:
> Java heap space
> Exception in thread "RMI TCP Connection(idle)"
> java.lang.OutOfMemoryError:
> Java heap space
> Exception in thread "RMI TCP Connection(idle)"
> java.lang.OutOfMemoryError:
> Java heap space
> Exception in thread "RMI TCP Connection(idle)"
> java.lang.OutOfMemoryError:
> Java heap space
> Exception in thread "InactivityMonitor WriteCheck"
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "RMI TCP Connection(idle)"
> java.lang.OutOfMemoryError:
> Java heap space
> Exception in thread "RMI TCP Connection(idle)"
> java.lang.OutOfMemoryError:
> Java heap space
> ...
>
>
> The stdout looks like this:
> 2009-01-28 10:48:35,664 [ueue://tracking] DEBUG
> vemq.store.amq.AMQMessageStore - Batch update done.
> 2009-01-28 10:48:33,537 [iveMQ Scheduler] DEBUG
> tore.amq.AMQPersistenceAdapter - lastDataFile: 5663
> 2009-01-28 10:48:27,177 [itor WriteCheck] DEBUG
> .transport.InactivityMonitor$2 - 72107 ms elapsed since last write
> check.
> 2009-01-28 10:48:56,877 [iveMQ Scheduler] DEBUG
> ha.impl.async.AsyncDataManager - lastFileId=5662, purgeList: (0) []
> 2009-01-28 10:48:52,634 [ueue://tracking] DEBUG
> vemq.store.amq.AMQMessageStore - Doing batch update... adding: 3
> removing: 4
> 2009-01-28 10:48:48,395 [127.0.0.1:39844] DEBUG
> vemq.store.amq.AMQMessageStore - Journalled message add for:
> ID:adbl006-40098-1233049753494-0:38:3395:1:2,
> at: offset = 168080, file = 5663, size = 384, type = 1
> 2009-01-28 10:48:48,395 [127.0.0.1:32929] DEBUG
> vemq.store.amq.AMQMessageStore - Journalled message remove for:
> ID:adbl006-40098-1233049753494-0:30:278402
> 8:1:1, at: offset = 167801, file = 5663, size = 279, type = 1
> 2009-01-28 10:48:37,784 [nitor ReadCheck] DEBUG
> .transport.InactivityMonitor$1 - 27573 ms elapsed since last read
> check.
> 2009-01-28 10:51:32,122 [127.0.0.1:39844] DEBUG
> vemq.store.amq.AMQMessageStore - Journalled message add for:
> ID:adbl006-40098-1233049753494-0:38:3392:1:2,
> at: offset = 168464, file = 5663, size = 384, type = 1
> 2009-01-28 10:51:27,528 [ueue://tracking] DEBUG
> vemq.store.amq.AMQMessageStore - Batch update done.
> 2009-01-28 10:50:48,614 [itor WriteCheck] DEBUG
> .transport.InactivityMonitor$2 - 165814 ms elapsed since last write
> check.
> 2009-01-28 10:50:48,614 [iveMQ Scheduler] DEBUG
> tore.amq.AMQPersistenceAdapter - dataFilesInProgress.values: (1)
> [{5663=3}]
> 2009-01-28 10:51:40,604 [ueue://tracking] DEBUG
> vemq.store.amq.AMQMessageStore - Doing batch update... adding: 2
> removing: 1
> 2009-01-28 10:51:36,365 [127.0.0.1:39844] DEBUG
> vemq.store.amq.AMQMessageStore - Journalled message add for:
> ID:adbl006-40098-1233049753494-0:38:3391:1:2,
> at: offset = 169127, file = 5663, size = 384, type = 1
> 2009-01-28 10:51:32,122 [127.0.0.1:32929] DEBUG
> vemq.store.amq.AMQMessageStore - Journalled message remove for:
> ID:adbl006-40098-1233049753494-0:30:278403
> 2:1:1, at: offset = 168848, file = 5663, size = 279, type = 1
> 2009-01-28 10:51:32,122 [nitor ReadCheck] DEBUG
> .transport.InactivityMonitor$1 - 197674 ms elapsed since last read
> check.
> 2009-01-28 10:55:28,803 [127.0.0.1:39844] DEBUG
> vemq.store.amq.AMQMessageStore - Journalled message add for:
> ID:adbl006-40098-1233049753494-0:38:3385:1:2,
> at: offset = 169511, file = 5663, size = 384, type = 1
> 2009-01-28 10:55:28,803 [ueue://tracking] DEBUG
> vemq.store.amq.AMQMessageStore - Batch update done.
> 2009-01-28 11:09:10,234 [.21.150.21:8161] ERROR
> rg.slf4j.impl.JCLLoggerAdapter - EXCEPTION
> java.lang.OutOfMemoryError: Java heap space
> 2009-01-28 10:58:33,322 [127.0.0.1:39844] DEBUG
> vemq.store.amq.AMQMessageStore - Journalled message add for:
> ID:adbl006-40098-1233049753494-0:38:3383:1:2,
> at: offset = 170174, file = 5663, size = 384, type = 1
> 2009-01-28 10:55:51,594 [127.0.0.1:32929] DEBUG
> vemq.store.amq.AMQMessageStore - Journalled message remove for:
> ID:adbl006-40098-1233049753494-0:30:278372
> 5:1:7, at: offset = 169895, file = 5663, size = 279, type = 1
> 2009-01-28 10:55:42,624 [nitor ReadCheck] DEBUG
> .transport.InactivityMonitor$1 - 424840 ms elapsed since last read
> check.
> 2009-01-28 11:17:46,299 [127.0.0.1:39844] DEBUG
> vemq.store.amq.AMQMessageStore - Journalled message add for:
> ID:adbl006-40098-1233049753494-0:38:2915:1:4,
> at: offset = 170558, file = 5663, size = 384, type = 1
> 2009-01-28 11:17:03,736 [127.0.0.1:32929] DEBUG
> vemq.store.amq.AMQMessageStore - Journalled message remove for:
> ID:adbl006-40098-1233049753494-0:30:278403
> 3:1:1, at: offset = 170942, file = 5663, size = 279, type = 1
> 2009-01-28 11:13:41,738 [3895909@qtp0-14] ERROR
> rg.slf4j.impl.JCLLoggerAdapter - /admin/queues.jsp
> java.lang.OutOfMemoryError: Java heap space
> 2009-01-28 11:09:25,197 [Timer-0 ] ERROR
> rg.slf4j.impl.JCLLoggerAdapter - Problem scavenging sessions
> java.lang.OutOfMemoryError: Java heap space
> 2009-01-28 11:19:30,007 [127.0.0.1:32929] DEBUG
> vemq.store.amq.AMQMessageStore - Journalled message remove for:
> ID:adbl006-40098-1233049753494-0:30:278403
> 5:1:1, at: offset = 171605, file = 5663, size = 279, type = 1
> 2009-01-28 11:18:50,504 [127.0.0.1:39844] DEBUG
> vemq.store.amq.AMQMessageStore - Journalled message add for:
> ID:adbl006-40098-1233049753494-0:38:3328:1:3,
> at: offset = 171221, file = 5663, size = 384, type = 1
> 2009-01-28 11:17:46,299 [nitor ReadCheck] DEBUG
> .transport.InactivityMonitor$1 - 1574177 ms elapsed since last read
> check.
> 2009-01-28 11:30:42,525 [127.0.0.1:32929] DEBUG
> emq.broker.TransportConnection - Transport failed:
> java.io.IOException:
> Unexpected error occured
> java.io.IOException: Unexpected error occured
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
> 192)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 2009-01-28 11:31:20,726 [127.0.0.1:39844] DEBUG
> emq.broker.TransportConnection - Transport failed:
> java.io.IOException:
> Unexpected error occured
> java.io.IOException: Unexpected error occured
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
> 192)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 2009-01-28 11:39:58,676 [127.0.0.1:39844] DEBUG
> emq.broker.TransportConnection - Stopping connection: /127.0.0.1:39844
> 2009-01-28 11:40:07,371 [127.0.0.1:32929] DEBUG
> emq.broker.TransportConnection - Stopping connection: /127.0.0.1:32929
> 2009-01-28 11:41:04,406 [127.0.0.1:39844] DEBUG
> emq.transport.tcp.TcpTransport - Stopping transport tcp:///
> 127.0.0.1:39844
> 2009-01-28 11:41:26,226 [127.0.0.1:32929] DEBUG
> emq.transport.tcp.TcpTransport - Stopping transport tcp:///
> 127.0.0.1:32929
> 2009-01-28 11:43:11,185 [127.0.0.1:39844] DEBUG
> emq.broker.TransportConnection - Stopped transport: /127.0.0.1:39844
> 2009-01-28 11:47:36,654 [127.0.0.1:32929] DEBUG
> emq.broker.TransportConnection - Stopped transport: /127.0.0.1:32929
> 2009-01-28 11:52:06,738 [127.0.0.1:32929] DEBUG
> emq.broker.TransportConnection - Cleaning up connection resources:
> /127.0.0.1:32929
> 2009-01-28 11:53:30,458 [127.0.0.1:32929] DEBUG
> q.broker.region.AbstractRegion - Removing consumer:
> ID:adbl006-40098-1233049753494-0:0:-1:1
> 2009-01-28 11:55:40,444 [127.0.0.1:32929] DEBUG
> q.broker.region.AbstractRegion - Removing consumer:
> ID:adbl006-40098-1233049753494-0:0:1:1
> 2009-01-28 11:51:49,128 [127.0.0.1:39844] DEBUG
> emq.broker.TransportConnection - Cleaning up connection resources:
> /127.0.0.1:39844
> 2009-01-28 11:56:07,668 [127.0.0.1:32929] WARN
> emq.broker.TransportConnection - Failed to remove consumer:
> ID:adbl006-40098-1233049753494-0:0:1:1. Reason
> : java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> 2009-01-28 11:56:23,041 [127.0.0.1:32929] DEBUG
> emq.broker.TransportConnection - Connection Stopped: /127.0.0.1:32929
> 2009-01-28 11:56:23,040 [127.0.0.1:39844] DEBUG
> q.broker.region.AbstractRegion - Removing consumer:
> ID:adbl006-40098-1233049753494-0:38:-1:1
> 2009-01-28 11:56:46,289 [127.0.0.1:39844] DEBUG
> emq.broker.TransportConnection - Connection Stopped: /127.0.0.1:39844
>
>
> Did anyone else detect this behavoir?
>
> Best regards
>
> Jörg
>
>
>
>
>
>
>
>
>
> --
> View this message in context: http://www.nabble.com/OutOfMemoryError-after-9505925-persistent-messages-tp21722796p21722796.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Haven't seen that before - could you raise an issue and create a junit
test case for us ?
cheers,
Rob
Rob Davies
http://fusesource.com
http://rajdavies.blogspot.com/