You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Ken Ringdahl <ke...@desktone.com> on 2007/10/26 17:30:53 UTC
ActiveMQ Transport Server OutOfMemoryError while running load test
I'm running a load test that intends to throw on the order of several
thousand connections at a 2 node broker system with failover (ActiveMQ
4.1.1) configured for tcp transport only. However, I'm just using a single
broker node right now and am getting out of memory errors at about 400
connections. I've set the Xmx to 1024M and the memory is only ~110 MB when
this error occurs in the log. I've pasted the exceptions below. The latter
2 exceptions do not immediately follow the first. They come maybe a minute
or so later. I've tried setting the preFetch to 1 in case it's keeping
messages in memory. But, to be honest, my test doesn't send messages until
all of the connections have been established. Any suggestions as to what
might be the problem here?
jvm 1 | Exception in thread "ActiveMQ Transport Server:
tcp://localhost:61616" java.lang.OutOfMemoryError: unable to create new
native thread
jvm 1 | at java.lang.Thread.start0(Native Method)
jvm 1 | at java.lang.Thread.start(Thread.java:574)
jvm 1 | at
org.apache.activemq.thread.DedicatedTaskRunner.<init>(DedicatedTaskRunner.java:45)
jvm 1 | at
org.apache.activemq.thread.TaskRunnerFactory.createTaskRunner(TaskRunnerFactory.java:77)
jvm 1 | at
org.apache.activemq.broker.TransportConnection.<init>(TransportConnection.java:174)
jvm 1 | at
org.apache.activemq.broker.jmx.ManagedTransportConnection.<init>(ManagedTransportConnection.java:55)
jvm 1 | at
org.apache.activemq.broker.jmx.ManagedTransportConnector.createConnection(ManagedTransportConnector.java:56)
jvm 1 | at
org.apache.activemq.broker.TransportConnector$1.onAccept(TransportConnector.java:147)
jvm 1 | at
org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:167)
jvm 1 | at java.lang.Thread.run(Thread.java:595)
jvm 1 | Exception in thread "RMI RenewClean-[172.16.105.110:36883]"
java.lang.OutOfMemoryError: unable to create new native thread
jvm 1 | at java.lang.Thread.start0(Native Method)
jvm 1 | at java.lang.Thread.start(Thread.java:574)
jvm 1 | at
sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:321)
jvm 1 | at sun.rmi.server.UnicastRef.free(UnicastRef.java:395)
jvm 1 | at sun.rmi.server.UnicastRef.done(UnicastRef.java:412)
jvm 1 | at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
jvm 1 | at
sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
jvm 1 | at
sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
jvm 1 | at
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
jvm 1 | at java.lang.Thread.run(Thread.java:595)
jvm 1 | Exception in thread "ActiveMQ Journal Checkpoint Worker"
java.lang.OutOfMemoryError: unable to create new native thread
jvm 1 | at java.lang.Thread.start0(Native Method)
jvm 1 | at java.lang.Thread.start(Thread.java:574)
jvm 1 | at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:429)
jvm 1 | at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:857)
jvm 1 | at
org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:376)
jvm 1 | at
org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
jvm 1 | at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
jvm 1 | at
org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
jvm 1 | at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
--
View this message in context: http://www.nabble.com/ActiveMQ-Transport-Server-OutOfMemoryError-while-running-load-test-tf4697790s2354.html#a13428819
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Transport Server OutOfMemoryError while running load
test
Posted by Dan Washusen <da...@reactive.org>.
Have you considered pooling the connections? You'll probably find the
performance of any Java app will degrade with that many threads...
Ken Ringdahl wrote:
>
> I'm running a load test that intends to throw on the order of several
> thousand connections at a 2 node broker system with failover (ActiveMQ
> 4.1.1) configured for tcp transport only. However, I'm just using a
> single broker node right now and am getting out of memory errors at about
> 400 connections. I've set the Xmx to 1024M and the memory is only ~110 MB
> when this error occurs in the log. I've pasted the exceptions below.
> The latter 2 exceptions do not immediately follow the first. They come
> maybe a minute or so later. I've tried setting the preFetch to 1 in case
> it's keeping messages in memory. But, to be honest, my test doesn't send
> messages until all of the connections have been established. Any
> suggestions as to what might be the problem here?
>
>
>
> jvm 1 | Exception in thread "ActiveMQ Transport Server:
> tcp://localhost:61616" java.lang.OutOfMemoryError: unable to create new
> native thread
> jvm 1 | at java.lang.Thread.start0(Native Method)
> jvm 1 | at java.lang.Thread.start(Thread.java:574)
> jvm 1 | at
> org.apache.activemq.thread.DedicatedTaskRunner.<init>(DedicatedTaskRunner.java:45)
> jvm 1 | at
> org.apache.activemq.thread.TaskRunnerFactory.createTaskRunner(TaskRunnerFactory.java:77)
> jvm 1 | at
> org.apache.activemq.broker.TransportConnection.<init>(TransportConnection.java:174)
> jvm 1 | at
> org.apache.activemq.broker.jmx.ManagedTransportConnection.<init>(ManagedTransportConnection.java:55)
> jvm 1 | at
> org.apache.activemq.broker.jmx.ManagedTransportConnector.createConnection(ManagedTransportConnector.java:56)
> jvm 1 | at
> org.apache.activemq.broker.TransportConnector$1.onAccept(TransportConnector.java:147)
> jvm 1 | at
> org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:167)
> jvm 1 | at java.lang.Thread.run(Thread.java:595)
> jvm 1 | Exception in thread "RMI RenewClean-[172.16.105.110:36883]"
> java.lang.OutOfMemoryError: unable to create new native thread
> jvm 1 | at java.lang.Thread.start0(Native Method)
> jvm 1 | at java.lang.Thread.start(Thread.java:574)
> jvm 1 | at
> sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:321)
> jvm 1 | at sun.rmi.server.UnicastRef.free(UnicastRef.java:395)
> jvm 1 | at sun.rmi.server.UnicastRef.done(UnicastRef.java:412)
> jvm 1 | at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
> jvm 1 | at
> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
> jvm 1 | at
> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
> jvm 1 | at
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
> jvm 1 | at java.lang.Thread.run(Thread.java:595)
> jvm 1 | Exception in thread "ActiveMQ Journal Checkpoint Worker"
> java.lang.OutOfMemoryError: unable to create new native thread
> jvm 1 | at java.lang.Thread.start0(Native Method)
> jvm 1 | at java.lang.Thread.start(Thread.java:574)
> jvm 1 | at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:429)
> jvm 1 | at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:857)
> jvm 1 | at
> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:376)
> jvm 1 | at
> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
> jvm 1 | at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
> jvm 1 | at
> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
> jvm 1 | at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
>
>
--
View this message in context: http://www.nabble.com/ActiveMQ-Transport-Server-OutOfMemoryError-while-running-load-test-tf4697790s2354.html#a13737559
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Transport Server OutOfMemoryError while running load
test
Posted by Ken Ringdahl <ke...@desktone.com>.
I should follow up with some of my results. I did resolve the OOM issue by
setting the thread size (-Xss) in the JVM to 128k. I was able to ramp up to
6k concurrent connections without any trouble at all. I was testing
scalability and fail over and was able to fail over these connections to
another ActiveMQ router in ~30 seconds. I did not push any higher than 6k
since that was far and away more than I would need, but it did seem that
there was significantly more headroom.
Hellweek wrote:
>
> For the past 2 weeks we have been testing ActiveMQ to answer this question
> ourself. Here is what we found.
>
> Creating a Connection takes one thread.
> Creating a Session takes one thread.
> Creating a NON Persisted Topic with a messageListner takes one thread,
> when the topic is no longer used the thread is returned to the system.
> Creating a Persisted Topic with a messageListner takes one thread this
> thread seems to stay around even when the Producer is disposed.
> Creating a QUEUE creates 1 thread per QUEUE when the QUEUE is disposed the
> thread servicing the QUEUE is still in use.
>
> On Windows 32 Bit we were able to get up to 2,000 threads before we could
> not create anymore.
> On Linux 32 Bit we were able to get up to about 2,500 threads before we
> could not create anymore. It is important to note that I had to make
> ulimit changes to memory size, virtual memory size and file handles in
> order to reach more then 1,000.
>
>
> On Windows 64 Bit with a 64 bit JVM we were able to get up to 5,000
> threads before we could not create anymore.
> On Linux 64 Bit with a 64bit JVM we were able to get up to about 6,000
> threads before we could not create anymore. It is important to note that
> I had to make ulimit changes to memory size, virtual memory size and file
> handles in order to reach more.
>
>
>
>
> Ken Ringdahl wrote:
>>
>> I'm running a load test that intends to throw on the order of several
>> thousand connections at a 2 node broker system with failover (ActiveMQ
>> 4.1.1) configured for tcp transport only. However, I'm just using a
>> single broker node right now and am getting out of memory errors at about
>> 400 connections. I've set the Xmx to 1024M and the memory is only ~110
>> MB when this error occurs in the log. I've pasted the exceptions below.
>> The latter 2 exceptions do not immediately follow the first. They come
>> maybe a minute or so later. I've tried setting the preFetch to 1 in case
>> it's keeping messages in memory. But, to be honest, my test doesn't send
>> messages until all of the connections have been established. Any
>> suggestions as to what might be the problem here?
>>
>>
>>
>> jvm 1 | Exception in thread "ActiveMQ Transport Server:
>> tcp://localhost:61616" java.lang.OutOfMemoryError: unable to create new
>> native thread
>> jvm 1 | at java.lang.Thread.start0(Native Method)
>> jvm 1 | at java.lang.Thread.start(Thread.java:574)
>> jvm 1 | at
>> org.apache.activemq.thread.DedicatedTaskRunner.<init>(DedicatedTaskRunner.java:45)
>> jvm 1 | at
>> org.apache.activemq.thread.TaskRunnerFactory.createTaskRunner(TaskRunnerFactory.java:77)
>> jvm 1 | at
>> org.apache.activemq.broker.TransportConnection.<init>(TransportConnection.java:174)
>> jvm 1 | at
>> org.apache.activemq.broker.jmx.ManagedTransportConnection.<init>(ManagedTransportConnection.java:55)
>> jvm 1 | at
>> org.apache.activemq.broker.jmx.ManagedTransportConnector.createConnection(ManagedTransportConnector.java:56)
>> jvm 1 | at
>> org.apache.activemq.broker.TransportConnector$1.onAccept(TransportConnector.java:147)
>> jvm 1 | at
>> org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:167)
>> jvm 1 | at java.lang.Thread.run(Thread.java:595)
>> jvm 1 | Exception in thread "RMI RenewClean-[172.16.105.110:36883]"
>> java.lang.OutOfMemoryError: unable to create new native thread
>> jvm 1 | at java.lang.Thread.start0(Native Method)
>> jvm 1 | at java.lang.Thread.start(Thread.java:574)
>> jvm 1 | at
>> sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:321)
>> jvm 1 | at sun.rmi.server.UnicastRef.free(UnicastRef.java:395)
>> jvm 1 | at sun.rmi.server.UnicastRef.done(UnicastRef.java:412)
>> jvm 1 | at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
>> jvm 1 | at
>> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
>> jvm 1 | at
>> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
>> jvm 1 | at
>> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
>> jvm 1 | at java.lang.Thread.run(Thread.java:595)
>> jvm 1 | Exception in thread "ActiveMQ Journal Checkpoint Worker"
>> java.lang.OutOfMemoryError: unable to create new native thread
>> jvm 1 | at java.lang.Thread.start0(Native Method)
>> jvm 1 | at java.lang.Thread.start(Thread.java:574)
>> jvm 1 | at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:429)
>> jvm 1 | at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:857)
>> jvm 1 | at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:376)
>> jvm 1 | at
>> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
>> jvm 1 | at
>> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
>> jvm 1 | at
>> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
>> jvm 1 | at
>> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
>>
>>
>
>
--
View this message in context: http://www.nabble.com/ActiveMQ-Transport-Server-OutOfMemoryError-while-running-load-test-tf4697790s2354.html#a13737723
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Transport Server OutOfMemoryError while running load
test
Posted by Hellweek <bw...@tfutures.com>.
For the past 2 weeks we have been testing ActiveMQ to answer this question
ourself. Here is what we found.
Creating a Connection takes one thread.
Creating a Session takes one thread.
Creating a NON Persisted Topic with a messageListner takes one thread, when
the topic is no longer used the thread is returned to the system.
Creating a Persisted Topic with a messageListner takes one thread this
thread seems to stay around even when the Producer is disposed.
Creating a QUEUE creates 1 thread per QUEUE when the QUEUE is disposed the
thread servicing the QUEUE is still in use.
On Windows 32 Bit we were able to get up to 2,000 threads before we could
not create anymore.
On Linux 32 Bit we were able to get up to about 2,500 threads before we
could not create anymore. It is important to note that I had to make ulimit
changes to memory size, virtual memory size and file handles in order to
reach more then 1,000.
On Windows 64 Bit with a 64 bit JVM we were able to get up to 5,000 threads
before we could not create anymore.
On Linux 64 Bit with a 64bit JVM we were able to get up to about 6,000
threads before we could not create anymore. It is important to note that I
had to make ulimit changes to memory size, virtual memory size and file
handles in order to reach more.
Ken Ringdahl wrote:
>
> I'm running a load test that intends to throw on the order of several
> thousand connections at a 2 node broker system with failover (ActiveMQ
> 4.1.1) configured for tcp transport only. However, I'm just using a
> single broker node right now and am getting out of memory errors at about
> 400 connections. I've set the Xmx to 1024M and the memory is only ~110 MB
> when this error occurs in the log. I've pasted the exceptions below.
> The latter 2 exceptions do not immediately follow the first. They come
> maybe a minute or so later. I've tried setting the preFetch to 1 in case
> it's keeping messages in memory. But, to be honest, my test doesn't send
> messages until all of the connections have been established. Any
> suggestions as to what might be the problem here?
>
>
>
> jvm 1 | Exception in thread "ActiveMQ Transport Server:
> tcp://localhost:61616" java.lang.OutOfMemoryError: unable to create new
> native thread
> jvm 1 | at java.lang.Thread.start0(Native Method)
> jvm 1 | at java.lang.Thread.start(Thread.java:574)
> jvm 1 | at
> org.apache.activemq.thread.DedicatedTaskRunner.<init>(DedicatedTaskRunner.java:45)
> jvm 1 | at
> org.apache.activemq.thread.TaskRunnerFactory.createTaskRunner(TaskRunnerFactory.java:77)
> jvm 1 | at
> org.apache.activemq.broker.TransportConnection.<init>(TransportConnection.java:174)
> jvm 1 | at
> org.apache.activemq.broker.jmx.ManagedTransportConnection.<init>(ManagedTransportConnection.java:55)
> jvm 1 | at
> org.apache.activemq.broker.jmx.ManagedTransportConnector.createConnection(ManagedTransportConnector.java:56)
> jvm 1 | at
> org.apache.activemq.broker.TransportConnector$1.onAccept(TransportConnector.java:147)
> jvm 1 | at
> org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:167)
> jvm 1 | at java.lang.Thread.run(Thread.java:595)
> jvm 1 | Exception in thread "RMI RenewClean-[172.16.105.110:36883]"
> java.lang.OutOfMemoryError: unable to create new native thread
> jvm 1 | at java.lang.Thread.start0(Native Method)
> jvm 1 | at java.lang.Thread.start(Thread.java:574)
> jvm 1 | at
> sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:321)
> jvm 1 | at sun.rmi.server.UnicastRef.free(UnicastRef.java:395)
> jvm 1 | at sun.rmi.server.UnicastRef.done(UnicastRef.java:412)
> jvm 1 | at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
> jvm 1 | at
> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
> jvm 1 | at
> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
> jvm 1 | at
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
> jvm 1 | at java.lang.Thread.run(Thread.java:595)
> jvm 1 | Exception in thread "ActiveMQ Journal Checkpoint Worker"
> java.lang.OutOfMemoryError: unable to create new native thread
> jvm 1 | at java.lang.Thread.start0(Native Method)
> jvm 1 | at java.lang.Thread.start(Thread.java:574)
> jvm 1 | at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:429)
> jvm 1 | at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:857)
> jvm 1 | at
> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:376)
> jvm 1 | at
> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
> jvm 1 | at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
> jvm 1 | at
> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
> jvm 1 | at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
>
>
--
View this message in context: http://www.nabble.com/ActiveMQ-Transport-Server-OutOfMemoryError-while-running-load-test-tf4697790s2354.html#a13733932
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Transport Server OutOfMemoryError while running load
test
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
if you look at the message
java.lang.OutOfMemoryError: unable to create new native thread
this will tell you that it has nothing to do with your Xmx setting.
Threads and thread stacks are allocated outside your Java heap (ie, your Xms and Xmx settings)
and you've simply run out of space to create new threads.
This could be memory related or just some configuration on your OS on how many threads you have set.
if it is memory related and if you have set -Xms to 1024m as well, lower it to something like 128m, then set your -Xss, to something low, maybe 128k
and try your test again.
Filip
Ken Ringdahl wrote:
> I'm running a load test that intends to throw on the order of several
> thousand connections at a 2 node broker system with failover (ActiveMQ
> 4.1.1) configured for tcp transport only. However, I'm just using a single
> broker node right now and am getting out of memory errors at about 400
> connections. I've set the Xmx to 1024M and the memory is only ~110 MB when
> this error occurs in the log. I've pasted the exceptions below. The latter
> 2 exceptions do not immediately follow the first. They come maybe a minute
> or so later. I've tried setting the preFetch to 1 in case it's keeping
> messages in memory. But, to be honest, my test doesn't send messages until
> all of the connections have been established. Any suggestions as to what
> might be the problem here?
>
>
>
> jvm 1 | Exception in thread "ActiveMQ Transport Server:
> tcp://localhost:61616" java.lang.OutOfMemoryError: unable to create new
> native thread
> jvm 1 | at java.lang.Thread.start0(Native Method)
> jvm 1 | at java.lang.Thread.start(Thread.java:574)
> jvm 1 | at
> org.apache.activemq.thread.DedicatedTaskRunner.<init>(DedicatedTaskRunner.java:45)
> jvm 1 | at
> org.apache.activemq.thread.TaskRunnerFactory.createTaskRunner(TaskRunnerFactory.java:77)
> jvm 1 | at
> org.apache.activemq.broker.TransportConnection.<init>(TransportConnection.java:174)
> jvm 1 | at
> org.apache.activemq.broker.jmx.ManagedTransportConnection.<init>(ManagedTransportConnection.java:55)
> jvm 1 | at
> org.apache.activemq.broker.jmx.ManagedTransportConnector.createConnection(ManagedTransportConnector.java:56)
> jvm 1 | at
> org.apache.activemq.broker.TransportConnector$1.onAccept(TransportConnector.java:147)
> jvm 1 | at
> org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:167)
> jvm 1 | at java.lang.Thread.run(Thread.java:595)
> jvm 1 | Exception in thread "RMI RenewClean-[172.16.105.110:36883]"
> java.lang.OutOfMemoryError: unable to create new native thread
> jvm 1 | at java.lang.Thread.start0(Native Method)
> jvm 1 | at java.lang.Thread.start(Thread.java:574)
> jvm 1 | at
> sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:321)
> jvm 1 | at sun.rmi.server.UnicastRef.free(UnicastRef.java:395)
> jvm 1 | at sun.rmi.server.UnicastRef.done(UnicastRef.java:412)
> jvm 1 | at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
> jvm 1 | at
> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
> jvm 1 | at
> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
> jvm 1 | at
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
> jvm 1 | at java.lang.Thread.run(Thread.java:595)
> jvm 1 | Exception in thread "ActiveMQ Journal Checkpoint Worker"
> java.lang.OutOfMemoryError: unable to create new native thread
> jvm 1 | at java.lang.Thread.start0(Native Method)
> jvm 1 | at java.lang.Thread.start(Thread.java:574)
> jvm 1 | at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:429)
> jvm 1 | at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:857)
> jvm 1 | at
> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:376)
> jvm 1 | at
> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
> jvm 1 | at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
> jvm 1 | at
> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
> jvm 1 | at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
>
>