You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by srmore <co...@gmail.com> on 2013/09/10 05:15:38 UTC
Error during startup - java.lang.OutOfMemoryError: unable to create
new native thread
I have a 5 node cluster with a load of around 300GB each. A node went down
and does not come up. I can see the following exception in the logs.
ERROR [main] 2013-09-09 21:50:56,117 AbstractCassandraDaemon.java (line
139) Fatal exception in thread Thread[main,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:703)
at
java.util.concurrent.ThreadPoolExecutor.prestartAllCoreThreads(ThreadPoolExecutor.java:1392)
at
org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:77)
at
org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:65)
at
org.apache.cassandra.concurrent.JMXConfigurableThreadPoolExecutor.<init>(JMXConfigurableThreadPoolExecutor.java:34)
at
org.apache.cassandra.concurrent.StageManager.multiThreadedConfigurableStage(StageManager.java:68)
at
org.apache.cassandra.concurrent.StageManager.<clinit>(StageManager.java:42)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:344)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:173)
The *ulimit -u* output is
*515042*
Which is far more than what is recommended [1] (10240) and I am skeptical
to set it to unlimited as recommended here [2]
Any pointers as to what could be the issue and how to get the node up.
[1]
http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docs&version=1.2&file=install/recommended_settings#cassandra/install/installRecommendSettings.html
[2]
http://mail-archives.apache.org/mod_mbox/cassandra-user/201303.mbox/%3CCAPqEvGE474Omea1BFLJ6U_pbAkOwWxk=Dwo35_pc-ATwB4_6iA@mail.gmail.com%3E
Thanks !
Re: Error during startup - java.lang.OutOfMemoryError: unable to
create new native thread
Posted by srmore <co...@gmail.com>.
Was too fast on the send button, sorry.
The thing I wanted to add was the
pending signals (-i) 515038
that looks odd to me, could that be related.
On Thu, Sep 19, 2013 at 4:53 PM, srmore <co...@gmail.com> wrote:
>
> I hit this issue again today and looks like changing -Xss option does not
> work :(
> I am on 1.0.11 (I know its old, we are upgrading to 1.2.9 right now) and
> have about 800-900GB of data. I can see cassandra is spending a lot of time
> reading the data files before it quits with "java.lang.OutOfMemoryError:
> unable to create new native thread" error.
>
> My hard and soft limits seems to be ok as well
> Datastax recommends [1]
>
> * soft nofile 32768
> * hard nofile 32768
>
>
> and I have
> hard nofile 65536
> soft nofile 65536
>
> My ulimit -u output is 515038 (which again should be sufficient)
>
> complete output
>
> ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 515038
> max locked memory (kbytes, -l) 32
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 10240
> cpu time (seconds, -t) unlimited
> max user processes (-u) 515038
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
>
>
>
> Has anyone run into this ?
>
> [1] http://www.datastax.com/docs/1.1/troubleshooting/index
>
> On Wed, Sep 11, 2013 at 8:47 AM, srmore <co...@gmail.com> wrote:
>
>> Thanks Viktor,
>>
>>
>> - check (cassandra-env.sh) -Xss size, you may need to increase it for
>> your JVM;
>>
>> This seems to have done the trick !
>>
>> Thanks !
>>
>>
>> On Tue, Sep 10, 2013 at 12:46 AM, Viktor Jevdokimov <
>> Viktor.Jevdokimov@adform.com> wrote:
>>
>>> For start:****
>>>
>>> - check (cassandra-env.sh) -Xss size, you may need to increase it for
>>> your JVM;****
>>>
>>> - check (cassandra-env.sh) -Xms and -Xmx size, you may need to increase
>>> it for your data load/bloom filter/index sizes.****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> ** **
>>> Best regards / Pagarbiai
>>> *Viktor Jevdokimov*
>>> Senior Developer
>>>
>>> [image: Adform News] <http://www.adform.com>
>>>
>>> *Visit us at Dmexco: *Hall 6 Stand B-52
>>> September 18-19 Cologne, Germany
>>> Email: Viktor.Jevdokimov@adform.com
>>> Phone: +370 5 212 3063, Fax +370 5 261 0453
>>> J. Jasinskio 16C, LT-03163 Vilnius, Lithuania
>>> Follow us on Twitter: @adforminsider<http://twitter.com/#!/adforminsider>
>>> Take a ride with Adform's Rich Media Suite<http://vimeo.com/adform/richmedia>
>>> [image: Dmexco 2013] <http://www.dmexco.de/>
>>>
>>> Disclaimer: The information contained in this message and attachments is
>>> intended solely for the attention and use of the named addressee and may be
>>> confidential. If you are not the intended recipient, you are reminded that
>>> the information remains the property of the sender. You must not use,
>>> disclose, distribute, copy, print or rely on this e-mail. If you have
>>> received this message in error, please contact the sender immediately and
>>> irrevocably delete this message and any copies.
>>>
>>> *From:* srmore [mailto:comomore@gmail.com]
>>> *Sent:* Tuesday, September 10, 2013 6:16 AM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Error during startup - java.lang.OutOfMemoryError: unable to
>>> create new native thread [heur]****
>>>
>>> ** **
>>>
>>>
>>> I have a 5 node cluster with a load of around 300GB each. A node went
>>> down and does not come up. I can see the following exception in the logs.
>>>
>>> ERROR [main] 2013-09-09 21:50:56,117 AbstractCassandraDaemon.java (line
>>> 139) Fatal exception in thread Thread[main,5,main]
>>> java.lang.OutOfMemoryError: unable to create new native thread
>>> at java.lang.Thread.start0(Native Method)
>>> at java.lang.Thread.start(Thread.java:640)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:703)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.prestartAllCoreThreads(ThreadPoolExecutor.java:1392)
>>> at
>>> org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:77)
>>> at
>>> org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:65)
>>> at
>>> org.apache.cassandra.concurrent.JMXConfigurableThreadPoolExecutor.<init>(JMXConfigurableThreadPoolExecutor.java:34)
>>> at
>>> org.apache.cassandra.concurrent.StageManager.multiThreadedConfigurableStage(StageManager.java:68)
>>> at
>>> org.apache.cassandra.concurrent.StageManager.<clinit>(StageManager.java:42)
>>> at
>>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:344)
>>> at
>>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:173)*
>>> ***
>>>
>>> ** **
>>>
>>> The *ulimit -u* output is
>>> *515042*****
>>>
>>> Which is far more than what is recommended [1] (10240) and I am
>>> skeptical to set it to unlimited as recommended here [2]****
>>>
>>> Any pointers as to what could be the issue and how to get the node up.**
>>> **
>>>
>>>
>>>
>>>
>>> [1]
>>> http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docs&version=1.2&file=install/recommended_settings#cassandra/install/installRecommendSettings.html
>>>
>>> [2]
>>> http://mail-archives.apache.org/mod_mbox/cassandra-user/201303.mbox/%3CCAPqEvGE474Omea1BFLJ6U_pbAkOwWxk=Dwo35_pc-ATwB4_6iA@mail.gmail.com%3E
>>> ****
>>>
>>> Thanks !****
>>>
>>
>>
>
Re: Error during startup - java.lang.OutOfMemoryError: unable to
create new native thread
Posted by srmore <co...@gmail.com>.
I hit this issue again today and looks like changing -Xss option does not
work :(
I am on 1.0.11 (I know its old, we are upgrading to 1.2.9 right now) and
have about 800-900GB of data. I can see cassandra is spending a lot of time
reading the data files before it quits with "java.lang.OutOfMemoryError:
unable to create new native thread" error.
My hard and soft limits seems to be ok as well
Datastax recommends [1]
* soft nofile 32768
* hard nofile 32768
and I have
hard nofile 65536
soft nofile 65536
My ulimit -u output is 515038 (which again should be sufficient)
complete output
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 515038
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 515038
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Has anyone run into this ?
[1] http://www.datastax.com/docs/1.1/troubleshooting/index
On Wed, Sep 11, 2013 at 8:47 AM, srmore <co...@gmail.com> wrote:
> Thanks Viktor,
>
>
> - check (cassandra-env.sh) -Xss size, you may need to increase it for your
> JVM;
>
> This seems to have done the trick !
>
> Thanks !
>
>
> On Tue, Sep 10, 2013 at 12:46 AM, Viktor Jevdokimov <
> Viktor.Jevdokimov@adform.com> wrote:
>
>> For start:****
>>
>> - check (cassandra-env.sh) -Xss size, you may need to increase it for
>> your JVM;****
>>
>> - check (cassandra-env.sh) -Xms and -Xmx size, you may need to increase
>> it for your data load/bloom filter/index sizes.****
>>
>> ** **
>>
>> ** **
>>
>> ** **
>> Best regards / Pagarbiai
>> *Viktor Jevdokimov*
>> Senior Developer
>>
>> [image: Adform News] <http://www.adform.com>
>>
>> *Visit us at Dmexco: *Hall 6 Stand B-52
>> September 18-19 Cologne, Germany
>> Email: Viktor.Jevdokimov@adform.com
>> Phone: +370 5 212 3063, Fax +370 5 261 0453
>> J. Jasinskio 16C, LT-03163 Vilnius, Lithuania
>> Follow us on Twitter: @adforminsider<http://twitter.com/#!/adforminsider>
>> Take a ride with Adform's Rich Media Suite<http://vimeo.com/adform/richmedia>
>> [image: Dmexco 2013] <http://www.dmexco.de/>
>>
>> Disclaimer: The information contained in this message and attachments is
>> intended solely for the attention and use of the named addressee and may be
>> confidential. If you are not the intended recipient, you are reminded that
>> the information remains the property of the sender. You must not use,
>> disclose, distribute, copy, print or rely on this e-mail. If you have
>> received this message in error, please contact the sender immediately and
>> irrevocably delete this message and any copies.
>>
>> *From:* srmore [mailto:comomore@gmail.com]
>> *Sent:* Tuesday, September 10, 2013 6:16 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Error during startup - java.lang.OutOfMemoryError: unable to
>> create new native thread [heur]****
>>
>> ** **
>>
>>
>> I have a 5 node cluster with a load of around 300GB each. A node went
>> down and does not come up. I can see the following exception in the logs.
>>
>> ERROR [main] 2013-09-09 21:50:56,117 AbstractCassandraDaemon.java (line
>> 139) Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: unable to create new native thread
>> at java.lang.Thread.start0(Native Method)
>> at java.lang.Thread.start(Thread.java:640)
>> at
>> java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:703)
>> at
>> java.util.concurrent.ThreadPoolExecutor.prestartAllCoreThreads(ThreadPoolExecutor.java:1392)
>> at
>> org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:77)
>> at
>> org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:65)
>> at
>> org.apache.cassandra.concurrent.JMXConfigurableThreadPoolExecutor.<init>(JMXConfigurableThreadPoolExecutor.java:34)
>> at
>> org.apache.cassandra.concurrent.StageManager.multiThreadedConfigurableStage(StageManager.java:68)
>> at
>> org.apache.cassandra.concurrent.StageManager.<clinit>(StageManager.java:42)
>> at
>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:344)
>> at
>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:173)**
>> **
>>
>> ** **
>>
>> The *ulimit -u* output is
>> *515042*****
>>
>> Which is far more than what is recommended [1] (10240) and I am skeptical
>> to set it to unlimited as recommended here [2]****
>>
>> Any pointers as to what could be the issue and how to get the node up.***
>> *
>>
>>
>>
>>
>> [1]
>> http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docs&version=1.2&file=install/recommended_settings#cassandra/install/installRecommendSettings.html
>>
>> [2]
>> http://mail-archives.apache.org/mod_mbox/cassandra-user/201303.mbox/%3CCAPqEvGE474Omea1BFLJ6U_pbAkOwWxk=Dwo35_pc-ATwB4_6iA@mail.gmail.com%3E
>> ****
>>
>> Thanks !****
>>
>
>
Re: Error during startup - java.lang.OutOfMemoryError: unable to
create new native thread
Posted by srmore <co...@gmail.com>.
Thanks Viktor,
- check (cassandra-env.sh) -Xss size, you may need to increase it for your
JVM;
This seems to have done the trick !
Thanks !
On Tue, Sep 10, 2013 at 12:46 AM, Viktor Jevdokimov <
Viktor.Jevdokimov@adform.com> wrote:
> For start:****
>
> - check (cassandra-env.sh) -Xss size, you may need to increase it for your
> JVM;****
>
> - check (cassandra-env.sh) -Xms and -Xmx size, you may need to increase it
> for your data load/bloom filter/index sizes.****
>
> ** **
>
> ** **
>
> ** **
> Best regards / Pagarbiai
> *Viktor Jevdokimov*
> Senior Developer
>
> [image: Adform News] <http://www.adform.com>
>
> *Visit us at Dmexco: *Hall 6 Stand B-52
> September 18-19 Cologne, Germany
> Email: Viktor.Jevdokimov@adform.com
> Phone: +370 5 212 3063, Fax +370 5 261 0453
> J. Jasinskio 16C, LT-03163 Vilnius, Lithuania
> Follow us on Twitter: @adforminsider <http://twitter.com/#!/adforminsider>
> Take a ride with Adform's Rich Media Suite<http://vimeo.com/adform/richmedia>
> [image: Dmexco 2013] <http://www.dmexco.de/>
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
> *From:* srmore [mailto:comomore@gmail.com]
> *Sent:* Tuesday, September 10, 2013 6:16 AM
> *To:* user@cassandra.apache.org
> *Subject:* Error during startup - java.lang.OutOfMemoryError: unable to
> create new native thread [heur]****
>
> ** **
>
>
> I have a 5 node cluster with a load of around 300GB each. A node went down
> and does not come up. I can see the following exception in the logs.
>
> ERROR [main] 2013-09-09 21:50:56,117 AbstractCassandraDaemon.java (line
> 139) Fatal exception in thread Thread[main,5,main]
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:640)
> at
> java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:703)
> at
> java.util.concurrent.ThreadPoolExecutor.prestartAllCoreThreads(ThreadPoolExecutor.java:1392)
> at
> org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:77)
> at
> org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:65)
> at
> org.apache.cassandra.concurrent.JMXConfigurableThreadPoolExecutor.<init>(JMXConfigurableThreadPoolExecutor.java:34)
> at
> org.apache.cassandra.concurrent.StageManager.multiThreadedConfigurableStage(StageManager.java:68)
> at
> org.apache.cassandra.concurrent.StageManager.<clinit>(StageManager.java:42)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:344)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:173)***
> *
>
> ** **
>
> The *ulimit -u* output is
> *515042*****
>
> Which is far more than what is recommended [1] (10240) and I am skeptical
> to set it to unlimited as recommended here [2]****
>
> Any pointers as to what could be the issue and how to get the node up.****
>
>
>
>
> [1]
> http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docs&version=1.2&file=install/recommended_settings#cassandra/install/installRecommendSettings.html
>
> [2]
> http://mail-archives.apache.org/mod_mbox/cassandra-user/201303.mbox/%3CCAPqEvGE474Omea1BFLJ6U_pbAkOwWxk=Dwo35_pc-ATwB4_6iA@mail.gmail.com%3E
> ****
>
> Thanks !****
>
RE: Error during startup - java.lang.OutOfMemoryError: unable to create new native thread
Posted by Viktor Jevdokimov <Vi...@adform.com>.
For start:
- check (cassandra-env.sh) -Xss size, you may need to increase it for your JVM;
- check (cassandra-env.sh) -Xms and -Xmx size, you may need to increase it for your data load/bloom filter/index sizes.
Best regards / Pagarbiai
Viktor Jevdokimov
Senior Developer
[Adform News] <http://www.adform.com>
Visit us at Dmexco: Hall 6 Stand B-52
September 18-19 Cologne, Germany
Email: Viktor.Jevdokimov@adform.com<ma...@adform.com>
Phone: +370 5 212 3063, Fax +370 5 261 0453
J. Jasinskio 16C, LT-03163 Vilnius, Lithuania
Follow us on Twitter: @adforminsider<http://twitter.com/#!/adforminsider>
Take a ride with Adform's Rich Media Suite<http://vimeo.com/adform/richmedia>
[Dmexco 2013] <http://www.dmexco.de/>
Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
From: srmore [mailto:comomore@gmail.com]
Sent: Tuesday, September 10, 2013 6:16 AM
To: user@cassandra.apache.org
Subject: Error during startup - java.lang.OutOfMemoryError: unable to create new native thread [heur]
I have a 5 node cluster with a load of around 300GB each. A node went down and does not come up. I can see the following exception in the logs.
ERROR [main] 2013-09-09 21:50:56,117 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[main,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:703)
at java.util.concurrent.ThreadPoolExecutor.prestartAllCoreThreads(ThreadPoolExecutor.java:1392)
at org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:77)
at org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.<init>(JMXEnabledThreadPoolExecutor.java:65)
at org.apache.cassandra.concurrent.JMXConfigurableThreadPoolExecutor.<init>(JMXConfigurableThreadPoolExecutor.java:34)
at org.apache.cassandra.concurrent.StageManager.multiThreadedConfigurableStage(StageManager.java:68)
at org.apache.cassandra.concurrent.StageManager.<clinit>(StageManager.java:42)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:344)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:173)
The ulimit -u output is
515042
Which is far more than what is recommended [1] (10240) and I am skeptical to set it to unlimited as recommended here [2]
Any pointers as to what could be the issue and how to get the node up.
[1] http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html?pagename=docs&version=1.2&file=install/recommended_settings#cassandra/install/installRecommendSettings.html
[2] http://mail-archives.apache.org/mod_mbox/cassandra-user/201303.mbox/%3CCAPqEvGE474Omea1BFLJ6U_pbAkOwWxk=Dwo35_pc-ATwB4_6iA@mail.gmail.com%3E
Thanks !