You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by xabhi <xa...@gmail.com> on 2018/07/02 11:17:26 UTC
java.lang.OutOfMemoryError: unable to create new native thread
Hi
I am using ActiveMQ v5.14.4 and have observed below multiple times now. I am
using NIO connector so thread count is not high (~203 threads when this
issue was happening).
Exception in thread "ActiveMQ InactivityMonitor WriteCheckTimer"
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
at
org.apache.activemq.transport.AbstractInactivityMonitor.writeCheck(AbstractInactivityMonitor.java:189)
at
org.apache.activemq.transport.AbstractInactivityMonitor$3.run(AbstractInactivityMonitor.java:153)
at
org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
which then triggers following warnings and no new clients are able to
connect after this.
[20180702 06:49:50.829 EDT (ActiveMQ NIO Worker 8)
org.apache.activemq.broker.TransportConnection#serviceTransportException 243
WARN] - Transport Connection to: tcp://10.219.31.188:34470 failed:
java.io.IOException: Timer already cancelled.
[20180702 06:49:50.945 EDT (ActiveMQ NIO Worker 7)
org.apache.activemq.broker.TransportConnection#serviceTransportException 243
WARN] - Transport Connection to: tcp://10.219.31.184:41305 failed:
java.io.IOException: Timer already cancelled.
[20180702 06:49:50.945 EDT (ActiveMQ NIO Worker 5)
org.apache.activemq.broker.TransportConnection#serviceTransportException 243
WARN] - Transport Connection to: tcp://10.219.31.189:41305 failed:
java.io.IOException: Timer already cancelled.
[20180702 06:49:50.945 EDT (ActiveMQ NIO Worker 3)
org.apache.activemq.broker.TransportConnection#serviceTransportException 243
WARN] - Transport Connection to: tcp://10.218.24.250:46430 failed:
java.io.IOException: Timer already cancelled.
What can cause this issue to happen? 203 threads doesn't seem too high. I
also found this related JIRA- https://issues.apache.org/jira/browse/AMQ-5543
<https://issues.apache.org/jira/browse/AMQ-5543> which seems to ahve fixed
this but i am still seeing this issue?
How should i go about fixing this? Does this look like a bug or a
configuration problem?
Thanks,
Abhi
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: java.lang.OutOfMemoryError: unable to create new native thread
Posted by xabhi <xa...@gmail.com>.
>>Are you sure about that -Xss1024m? That would set the thread stack size to
a
monstrous 1GB. Perhaps you wanted to configure the initial heap size (-Xms)
there but made a typo?
Thanks for the help. The framework that launches the Java application was
incorrectly setting default stack size to 1024m.
Thanks,
Abhi
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: java.lang.OutOfMemoryError: unable to create new native thread
Posted by ho...@post.cz.
Hello,
> The JVM flags used are: ... -Xss1024m
Are you sure about that -Xss1024m? That would set the thread stack size to a
monstrous 1GB. Perhaps you wanted to configure the initial heap size (-Xms)
there but made a typo? If that's really what's configured there try to
remove it or modify it to -Xms1024m which is probably what was intended to
be set there. That could be all you need to do.
With 64-bit JVM you shouldn't be normally hitting the native memory
exhaustion issue unless the whole system is low on total memory available.
It was more common issue with 32-bit systems given the limited address
space.
But check the amount of RAM and monitor the memory usage (on the whole
system) as well, maybe you're operating close to the maximum available. In
such case you can either add more RAM to the system or decrease the maximum
heap size (Xmx) if that's acceptable (you can do that with another instances
as well - it's about lowering the total memory requirements). Or if there's
no other way, you can still try lowering the stack size a little - your
default is 1MB, so you may try something in the range: 768k - 896k.
Max processes limit looks high enough so that shouldn't be an issue.
You can monitor the count of threads for the user to verify that the limit
isn't being sometimes hit, something like the following example should be
usable (there might be better alternatives):
ps h -fLu username | wc -l
Best regards,
Petr
"It's at 64-bit system/JVM.
The process limits from /proc/pid/limits are:
Max processes 16341 16341
processes
Max open files 65536 65536 files
The JVM flags used are : -Xmx8g -Xss1024m
The default value in JVM is:
amqstg@stokes31.toa(nyc):~> java -XX:+PrintFlagsFinal -version | grep
ThreadStackSize
intx ThreadStackSize = 1024
{pd product}
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
The OS limits and heap sizes seem more that enough to me. I have another
instance that runs with same configuration and hasn't exhibited this
behavior. What other factors should i look at for more information next time
this issue occurs.
How can i increase the native memory space if thats what is causing this?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
"
Re: java.lang.OutOfMemoryError: unable to create new native thread
Posted by xabhi <xa...@gmail.com>.
It's at 64-bit system/JVM.
The process limits from /proc/pid/limits are:
Max processes 16341 16341
processes
Max open files 65536 65536 files
The JVM flags used are : -Xmx8g -Xss1024m
The default value in JVM is:
amqstg@stokes31.toa(nyc):~> java -XX:+PrintFlagsFinal -version | grep
ThreadStackSize
intx ThreadStackSize = 1024
{pd product}
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
The OS limits and heap sizes seem more that enough to me. I have another
instance that runs with same configuration and hasn't exhibited this
behavior. What other factors should i look at for more information next time
this issue occurs.
How can i increase the native memory space if thats what is causing this?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: java.lang.OutOfMemoryError: unable to create new native thread
Posted by ho...@post.cz.
Hello,
there are various reasons why this may happen, but with your low amount of
threads - are you by any chance running this at 32-bit OS or at least JVM?
If yes, it isn't that difficult to achieve that with large heap sizes when
you're operating near the address space limitations.
In such case you can:
- Decrease the heap size (-Xmx) a little in order to leave more room for the
native memory purposes. I think that even decreasing the MaxPermSize or
MaxMetaspaceSize (Java 8+) might work, but I'm not really sure about the
metaspace.
- Decrease the thread stack size (-Xss) a little to reduce the per-thread
native memory consumption.
Both settings would need to be adjusted sensibly as lowering them too much
would have other negative effects (performance, OutOfMemoryError on heap
space, StackOverflowError).
As for the thread stack size you can determine the default value via the -
XX:+PrintFlagsFinal JVM option and then reduce it slightly, example:
# /opt/jre1.8.0/bin/java -XX:+PrintFlagsFinal -version 2>&1|grep StackSize
...
intx ThreadStackSize = 320
{pd product}
...
In this case I'd start with setting it to 256k, e.g. -Xss256k
If it's at 64-bit system/JVM then that would be more interesting and in such
case I'd start with reviewing the OS limits (at Linux it would be the
maximum user processes limit that you can review by executing "ulimit -u"
(soft limit) and "ulimit -Hu" (hard limit) under the user running your JVM
process). But with those claimed ~200 threads you shouldn't be hitting even
the default limit I'd say. Maybe if you'd be running many concurrent
instances...
Best regards,
Petr
"Hi
I am using ActiveMQ v5.14.4 and have observed below multiple times now. I am
using NIO connector so thread count is not high (~203 threads when this
issue was happening).
Exception in thread "ActiveMQ InactivityMonitor WriteCheckTimer"
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:
950)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
1368)
at
org.apache.activemq.transport.AbstractInactivityMonitor.writeCheck
(AbstractInactivityMonitor.java:189)
at
org.apache.activemq.transport.AbstractInactivityMonitor$3.run
(AbstractInactivityMonitor.java:153)
at
org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:
33)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
which then triggers following warnings and no new clients are able to
connect after this.
[20180702 06:49:50.829 EDT (ActiveMQ NIO Worker 8)
org.apache.activemq.broker.TransportConnection#serviceTransportException 243
WARN] - Transport Connection to: tcp://10.219.31.188:34470 failed:
java.io.IOException: Timer already cancelled.
[20180702 06:49:50.945 EDT (ActiveMQ NIO Worker 7)
org.apache.activemq.broker.TransportConnection#serviceTransportException 243
WARN] - Transport Connection to: tcp://10.219.31.184:41305 failed:
java.io.IOException: Timer already cancelled.
[20180702 06:49:50.945 EDT (ActiveMQ NIO Worker 5)
org.apache.activemq.broker.TransportConnection#serviceTransportException 243
WARN] - Transport Connection to: tcp://10.219.31.189:41305 failed:
java.io.IOException: Timer already cancelled.
[20180702 06:49:50.945 EDT (ActiveMQ NIO Worker 3)
org.apache.activemq.broker.TransportConnection#serviceTransportException 243
WARN] - Transport Connection to: tcp://10.218.24.250:46430 failed:
java.io.IOException: Timer already cancelled.
What can cause this issue to happen? 203 threads doesn't seem too high. I
also found this related JIRA- https://issues.apache.org/jira/browse/AMQ-5543
<https://issues.apache.org/jira/browse/AMQ-5543> which seems to ahve fixed
this but i am still seeing this issue?
How should i go about fixing this? Does this look like a bug or a
configuration problem?
Thanks,
Abhi
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
"