You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jason Wee <pe...@gmail.com> on 2014/11/10 15:25:41 UTC
What actually causing java.lang.OutOfMemoryError: unable to create
new native thread
Hello people, below is an extraction from cassandra system log.
ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[Thread-273,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
at
org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)
Investigated into the call until the java native call,
http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698
if (native_thread->osthread() == NULL) {
// No one should hold a reference to the 'native_thread'.
delete native_thread;
if (JvmtiExport::should_post_resource_exhausted()) {
JvmtiExport::post_resource_exhausted(
JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR |
JVMTI_RESOURCE_EXHAUSTED_THREADS,
"unable to create new native thread");
}
THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
"unable to create new native thread");
}
Question. Is that out of memory error due to native os memory or java heap?
Stacked size to the jvm is -Xss128k. Operating system file descriptor max
user processes 260000. open files capped at 65536
Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR
and JVMTI_RESOURCE_EXHAUSTED_THREADS means too?
Thank you.
Jason
Re: What actually causing java.lang.OutOfMemoryError: unable to
create new native thread
Posted by Jason Wee <pe...@gmail.com>.
Hi, thank you for response.
using 64bit and kernel 2.6.32-358.18.1.el6.x86_64.
# cat /proc/13405/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 260000 260000
processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 255762 255762 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Is the stack size and / or Max open files is sufficient? The rest fd pretty
much infinite.
Jason
On Tue, Nov 11, 2014 at 4:09 AM, Chris Lohfink <cl...@gmail.com> wrote:
> if your using 64 bit, check output of:
>
> cat /proc/{cassandra pid}/limits
>
> some older linux kernels wont work with above so if it doesnt exist check
> the ulimit -a output for the cassandra user. max processes per user may
> be your issue as well.
>
> ---
> Chris Lohfink
>
>
> On Mon, Nov 10, 2014 at 11:21 AM, graham sanderson <gr...@vast.com>
> wrote:
>
>> First question are you running 32bit or 64bit… on 32bit you can easily
>> run out of virtual address space for thread stacks.
>>
>> On Nov 10, 2014, at 8:25 AM, Jason Wee <pe...@gmail.com> wrote:
>>
>> Hello people, below is an extraction from cassandra system log.
>>
>> ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java
>> (line 139) Fatal exception in thread Thread[Thread-273,5,main]
>> java.lang.OutOfMemoryError: unable to create new native thread
>> at java.lang.Thread.start0(Native Method)
>> at java.lang.Thread.start(Thread.java:640)
>> at
>> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
>> at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
>> at
>> org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
>> at
>> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)
>>
>> Investigated into the call until the java native call,
>> http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698
>>
>> if (native_thread->osthread() == NULL) {
>> // No one should hold a reference to the 'native_thread'.
>> delete native_thread;
>> if (JvmtiExport::should_post_resource_exhausted()) {
>> JvmtiExport::post_resource_exhausted(
>> JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR |
>> JVMTI_RESOURCE_EXHAUSTED_THREADS,
>> "unable to create new native thread");
>> }
>> THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
>> "unable to create new native thread");
>> }
>>
>> Question. Is that out of memory error due to native os memory or java
>> heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor
>> max user processes 260000. open files capped at 65536
>>
>> Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR
>> and JVMTI_RESOURCE_EXHAUSTED_THREADS means too?
>>
>> Thank you.
>>
>> Jason
>>
>>
>>
>
Re: What actually causing java.lang.OutOfMemoryError: unable to
create new native thread
Posted by Chris Lohfink <cl...@gmail.com>.
if your using 64 bit, check output of:
cat /proc/{cassandra pid}/limits
some older linux kernels wont work with above so if it doesnt exist check
the ulimit -a output for the cassandra user. max processes per user may be
your issue as well.
---
Chris Lohfink
On Mon, Nov 10, 2014 at 11:21 AM, graham sanderson <gr...@vast.com> wrote:
> First question are you running 32bit or 64bit… on 32bit you can easily run
> out of virtual address space for thread stacks.
>
> On Nov 10, 2014, at 8:25 AM, Jason Wee <pe...@gmail.com> wrote:
>
> Hello people, below is an extraction from cassandra system log.
>
> ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java
> (line 139) Fatal exception in thread Thread[Thread-273,5,main]
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:640)
> at
> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
> at
> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)
>
> Investigated into the call until the java native call,
> http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698
>
> if (native_thread->osthread() == NULL) {
> // No one should hold a reference to the 'native_thread'.
> delete native_thread;
> if (JvmtiExport::should_post_resource_exhausted()) {
> JvmtiExport::post_resource_exhausted(
> JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR |
> JVMTI_RESOURCE_EXHAUSTED_THREADS,
> "unable to create new native thread");
> }
> THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
> "unable to create new native thread");
> }
>
> Question. Is that out of memory error due to native os memory or java
> heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor
> max user processes 260000. open files capped at 65536
>
> Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR
> and JVMTI_RESOURCE_EXHAUSTED_THREADS means too?
>
> Thank you.
>
> Jason
>
>
>
Re: What actually causing java.lang.OutOfMemoryError: unable to create new native thread
Posted by graham sanderson <gr...@vast.com>.
First question are you running 32bit or 64bit… on 32bit you can easily run out of virtual address space for thread stacks.
> On Nov 10, 2014, at 8:25 AM, Jason Wee <pe...@gmail.com> wrote:
>
> Hello people, below is an extraction from cassandra system log.
>
> ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-273,5,main]
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:640)
> at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
> at org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
> at org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)
>
> Investigated into the call until the java native call, http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698 <http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698>
>
> if (native_thread->osthread() == NULL) {
> // No one should hold a reference to the 'native_thread'.
> delete native_thread;
> if (JvmtiExport::should_post_resource_exhausted()) {
> JvmtiExport::post_resource_exhausted(
> JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR | JVMTI_RESOURCE_EXHAUSTED_THREADS,
> "unable to create new native thread");
> }
> THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
> "unable to create new native thread");
> }
>
> Question. Is that out of memory error due to native os memory or java heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor max user processes 260000. open files capped at 65536
>
> Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR and JVMTI_RESOURCE_EXHAUSTED_THREADS means too?
>
> Thank you.
>
> Jason