You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Yatong Zhang <bl...@gmail.com> on 2014/05/05 02:58:20 UTC
Cassandra 2.0.7 always failes due to 'too may open files' error
Hi there,
After I changed compaction strategy to 'leveled', one of my nodes keeps
reporting 'too many open files. But I have done some configuration
following: http://www.datastax.com/docs/1.1/install/recommended_settingsand
http://www.datastax.com/docs/1.1/troubleshooting/index#toomany
I am using 2.0.7 and here is the system info:
[root@storage5 ~]# ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 126550
> max locked memory (kbytes, -l) unlimited
> max memory size (kbytes, -m) unlimited
> open files (-n) 999999
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 10240
> cpu time (seconds, -t) unlimited
> max user processes (-u) 126550
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> [root@storage5 ~]# sysctl -a | grep fs.file-max
> fs.file-max = 9999999
> [root@storage5 ~]# lsof -n | grep java | wc -l
> 5103
> [root@storage5 ~]# lsof | wc -l
> 6567
>
Here is the error:
> ERROR [CompactionExecutor:1] 2014-05-05 08:33:56,077 CassandraDaemon.java
> (line 198) Exception in thread Thread[CompactionExecutor:1,1,main]
> FSWriteError in /data6/cass/mydb/images/mydb-images-tmp-jb-109136-Filter.db
> at
> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475)
> at
> org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.io.FileNotFoundException:
> /data6/cass/mydb/images/mydb-images-tmp-jb-109136-Filter.db (Too many open
> files)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:110)
> at
> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:466)
> ... 13 more
> ERROR [CompactionExecutor:1] 2014-05-05 08:33:56,105 StorageService.java
> (line 367) Stopping gossiper
>
Re: Cassandra 2.0.7 always failes due to 'too may open files' error
Posted by Yatong Zhang <bl...@gmail.com>.
Yes the global limits are OK. I added cassandra to '/etc/rc.local' to make
it auto-startup, but seems the modification of limits didn't take effect. I
observed this as Bryan suggested, so I added
ulimit -SHn 999999
>
to '/etc/rc.local' and before cassandra start command, and it worked.
On Thu, May 8, 2014 at 3:34 AM, Nikolay Mihaylov <nm...@nmmm.nu> wrote:
> sorry, probably somebody mentioned it, but did you checked global limit?
>
> cat /proc/sys/fs/file-max
> cat /proc/sys/fs/file-nr
>
>
>
> On Mon, May 5, 2014 at 10:31 PM, Bryan Talbot <br...@playnext.com>wrote:
>
>> Running
>>
>> #> cat /proc/$(cat /var/run/cassandra.pid)/limits
>>
>> as root or your cassandra user will tell you what limits it's actually
>> running with.
>>
>>
>>
>>
>> On Sun, May 4, 2014 at 10:12 PM, Yatong Zhang <bl...@gmail.com>wrote:
>>
>>> I am running 'repair' when the error occurred. And just a few days
>>> before I changed the compaction strategy to 'leveled'. don know if this
>>> helps
>>>
>>>
>>> On Mon, May 5, 2014 at 1:10 PM, Yatong Zhang <bl...@gmail.com>wrote:
>>>
>>>> Cassandra is running as root
>>>>
>>>> [root@storage5 ~]# ps aux | grep java
>>>>> root 1893 42.0 24.0 7630664 3904000 ? Sl 10:43 60:01 java
>>>>> -ea -javaagent:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar
>>>>> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities
>>>>> -XX:ThreadPriorityPolicy=42 -Xms3959M -Xmx3959M -Xmn400M
>>>>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003
>>>>> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>>>>> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
>>>>> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
>>>>> -XX:+UseTLAB -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true
>>>>> -Dcom.sun.management.jmxremote.port=7199
>>>>> -Dcom.sun.management.jmxremote.ssl=false
>>>>> -Dcom.sun.management.jmxremote.authenticate=false
>>>>> -Dlog4j.configuration=log4j-server.properties
>>>>> -Dlog4j.defaultInitOverride=true -Dcassandra-pidfile=/var/run/cassandra.pid
>>>>> -cp
>>>>> /mydb/cassandra/bin/../conf:/mydb/cassandra/bin/../build/classes/main:/mydb/cassandra/bin/../build/classes/thrift:/mydb/cassandra/bin/../lib/antlr-3.2.jar:/mydb/cassandra/bin/../lib/apache-cassandra-2.0.7.jar:/mydb/cassandra/bin/../lib/apache-cassandra-clientutil-2.0.7.jar:/mydb/cassandra/bin/../lib/apache-cassandra-thrift-2.0.7.jar:/mydb/cassandra/bin/../lib/commons-cli-1.1.jar:/mydb/cassandra/bin/../lib/commons-codec-1.2.jar:/mydb/cassandra/bin/../lib/commons-lang3-3.1.jar:/mydb/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/mydb/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/mydb/cassandra/bin/../lib/disruptor-3.0.1.jar:/mydb/cassandra/bin/../lib/guava-15.0.jar:/mydb/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/mydb/cassandra/bin/../lib/jackson-core-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jackson-mapper-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar:/mydb/cassandra/bin/../lib/jbcrypt-0.3m.jar:/mydb/cassandra/bin/../lib/jline-1.0.jar:/mydb/cassandra/bin/../lib/json-simple-1.1.jar:/mydb/cassandra/bin/../lib/libthrift-0.9.1.jar:/mydb/cassandra/bin/../lib/log4j-1.2.16.jar:/mydb/cassandra/bin/../lib/lz4-1.2.0.jar:/mydb/cassandra/bin/../lib/metrics-core-2.2.0.jar:/mydb/cassandra/bin/../lib/netty-3.6.6.Final.jar:/mydb/cassandra/bin/../lib/reporter-config-2.1.0.jar:/mydb/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/mydb/cassandra/bin/../lib/slf4j-api-1.7.2.jar:/mydb/cassandra/bin/../lib/slf4j-log4j12-1.7.2.jar:/mydb/cassandra/bin/../lib/snakeyaml-1.11.jar:/mydb/cassandra/bin/../lib/snappy-java-1.0.5.jar:/mydb/cassandra/bin/../lib/snaptree-0.1.jar:/mydb/cassandra/bin/../lib/super-csv-2.1.0.jar:/mydb/cassandra/bin/../lib/thrift-server-0.3.3.jar
>>>>> org.apache.cassandra.service.CassandraDaemon
>>>>>
>>>>
>>>>
>>>>
>>>> On Mon, May 5, 2014 at 1:02 PM, Philip Persad <ph...@gmail.com>wrote:
>>>>
>>>>> Have you tried running "ulimit -a" as the Cassandra user instead of
>>>>> as root? It is possible that your configured a high file limit for root but
>>>>> not for the user running the Cassandra process.
>>>>>
>>>>>
>>>>> On Sun, May 4, 2014 at 6:07 PM, Yatong Zhang <bl...@gmail.com>wrote:
>>>>>
>>>>>> [root@storage5 ~]# lsof -n | grep java | wc -l
>>>>>>> 5103
>>>>>>> [root@storage5 ~]# lsof | wc -l
>>>>>>> 6567
>>>>>>
>>>>>>
>>>>>> It's mentioned in previous mail:)
>>>>>>
>>>>>>
>>>>>> On Mon, May 5, 2014 at 9:03 AM, nash <na...@gmail.com> wrote:
>>>>>>
>>>>>>> The lsof command or /proc can tell you how many open files it has.
>>>>>>> How many is it?
>>>>>>>
>>>>>>> --nash
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
Re: Cassandra 2.0.7 always failes due to 'too may open files' error
Posted by Nikolay Mihaylov <nm...@nmmm.nu>.
sorry, probably somebody mentioned it, but did you checked global limit?
cat /proc/sys/fs/file-max
cat /proc/sys/fs/file-nr
On Mon, May 5, 2014 at 10:31 PM, Bryan Talbot <br...@playnext.com>wrote:
> Running
>
> #> cat /proc/$(cat /var/run/cassandra.pid)/limits
>
> as root or your cassandra user will tell you what limits it's actually
> running with.
>
>
>
>
> On Sun, May 4, 2014 at 10:12 PM, Yatong Zhang <bl...@gmail.com> wrote:
>
>> I am running 'repair' when the error occurred. And just a few days before
>> I changed the compaction strategy to 'leveled'. don know if this helps
>>
>>
>> On Mon, May 5, 2014 at 1:10 PM, Yatong Zhang <bl...@gmail.com> wrote:
>>
>>> Cassandra is running as root
>>>
>>> [root@storage5 ~]# ps aux | grep java
>>>> root 1893 42.0 24.0 7630664 3904000 ? Sl 10:43 60:01 java
>>>> -ea -javaagent:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar
>>>> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities
>>>> -XX:ThreadPriorityPolicy=42 -Xms3959M -Xmx3959M -Xmn400M
>>>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003
>>>> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>>>> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
>>>> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
>>>> -XX:+UseTLAB -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true
>>>> -Dcom.sun.management.jmxremote.port=7199
>>>> -Dcom.sun.management.jmxremote.ssl=false
>>>> -Dcom.sun.management.jmxremote.authenticate=false
>>>> -Dlog4j.configuration=log4j-server.properties
>>>> -Dlog4j.defaultInitOverride=true -Dcassandra-pidfile=/var/run/cassandra.pid
>>>> -cp
>>>> /mydb/cassandra/bin/../conf:/mydb/cassandra/bin/../build/classes/main:/mydb/cassandra/bin/../build/classes/thrift:/mydb/cassandra/bin/../lib/antlr-3.2.jar:/mydb/cassandra/bin/../lib/apache-cassandra-2.0.7.jar:/mydb/cassandra/bin/../lib/apache-cassandra-clientutil-2.0.7.jar:/mydb/cassandra/bin/../lib/apache-cassandra-thrift-2.0.7.jar:/mydb/cassandra/bin/../lib/commons-cli-1.1.jar:/mydb/cassandra/bin/../lib/commons-codec-1.2.jar:/mydb/cassandra/bin/../lib/commons-lang3-3.1.jar:/mydb/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/mydb/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/mydb/cassandra/bin/../lib/disruptor-3.0.1.jar:/mydb/cassandra/bin/../lib/guava-15.0.jar:/mydb/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/mydb/cassandra/bin/../lib/jackson-core-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jackson-mapper-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar:/mydb/cassandra/bin/../lib/jbcrypt-0.3m.jar:/mydb/cassandra/bin/../lib/jline-1.0.jar:/mydb/cassandra/bin/../lib/json-simple-1.1.jar:/mydb/cassandra/bin/../lib/libthrift-0.9.1.jar:/mydb/cassandra/bin/../lib/log4j-1.2.16.jar:/mydb/cassandra/bin/../lib/lz4-1.2.0.jar:/mydb/cassandra/bin/../lib/metrics-core-2.2.0.jar:/mydb/cassandra/bin/../lib/netty-3.6.6.Final.jar:/mydb/cassandra/bin/../lib/reporter-config-2.1.0.jar:/mydb/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/mydb/cassandra/bin/../lib/slf4j-api-1.7.2.jar:/mydb/cassandra/bin/../lib/slf4j-log4j12-1.7.2.jar:/mydb/cassandra/bin/../lib/snakeyaml-1.11.jar:/mydb/cassandra/bin/../lib/snappy-java-1.0.5.jar:/mydb/cassandra/bin/../lib/snaptree-0.1.jar:/mydb/cassandra/bin/../lib/super-csv-2.1.0.jar:/mydb/cassandra/bin/../lib/thrift-server-0.3.3.jar
>>>> org.apache.cassandra.service.CassandraDaemon
>>>>
>>>
>>>
>>>
>>> On Mon, May 5, 2014 at 1:02 PM, Philip Persad <ph...@gmail.com>wrote:
>>>
>>>> Have you tried running "ulimit -a" as the Cassandra user instead of as
>>>> root? It is possible that your configured a high file limit for root but
>>>> not for the user running the Cassandra process.
>>>>
>>>>
>>>> On Sun, May 4, 2014 at 6:07 PM, Yatong Zhang <bl...@gmail.com>wrote:
>>>>
>>>>> [root@storage5 ~]# lsof -n | grep java | wc -l
>>>>>> 5103
>>>>>> [root@storage5 ~]# lsof | wc -l
>>>>>> 6567
>>>>>
>>>>>
>>>>> It's mentioned in previous mail:)
>>>>>
>>>>>
>>>>> On Mon, May 5, 2014 at 9:03 AM, nash <na...@gmail.com> wrote:
>>>>>
>>>>>> The lsof command or /proc can tell you how many open files it has.
>>>>>> How many is it?
>>>>>>
>>>>>> --nash
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
>
Re: Cassandra 2.0.7 always failes due to 'too may open files' error
Posted by Bryan Talbot <br...@playnext.com>.
Running
#> cat /proc/$(cat /var/run/cassandra.pid)/limits
as root or your cassandra user will tell you what limits it's actually
running with.
On Sun, May 4, 2014 at 10:12 PM, Yatong Zhang <bl...@gmail.com> wrote:
> I am running 'repair' when the error occurred. And just a few days before
> I changed the compaction strategy to 'leveled'. don know if this helps
>
>
> On Mon, May 5, 2014 at 1:10 PM, Yatong Zhang <bl...@gmail.com> wrote:
>
>> Cassandra is running as root
>>
>> [root@storage5 ~]# ps aux | grep java
>>> root 1893 42.0 24.0 7630664 3904000 ? Sl 10:43 60:01 java
>>> -ea -javaagent:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar
>>> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities
>>> -XX:ThreadPriorityPolicy=42 -Xms3959M -Xmx3959M -Xmn400M
>>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003
>>> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>>> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
>>> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
>>> -XX:+UseTLAB -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true
>>> -Dcom.sun.management.jmxremote.port=7199
>>> -Dcom.sun.management.jmxremote.ssl=false
>>> -Dcom.sun.management.jmxremote.authenticate=false
>>> -Dlog4j.configuration=log4j-server.properties
>>> -Dlog4j.defaultInitOverride=true -Dcassandra-pidfile=/var/run/cassandra.pid
>>> -cp
>>> /mydb/cassandra/bin/../conf:/mydb/cassandra/bin/../build/classes/main:/mydb/cassandra/bin/../build/classes/thrift:/mydb/cassandra/bin/../lib/antlr-3.2.jar:/mydb/cassandra/bin/../lib/apache-cassandra-2.0.7.jar:/mydb/cassandra/bin/../lib/apache-cassandra-clientutil-2.0.7.jar:/mydb/cassandra/bin/../lib/apache-cassandra-thrift-2.0.7.jar:/mydb/cassandra/bin/../lib/commons-cli-1.1.jar:/mydb/cassandra/bin/../lib/commons-codec-1.2.jar:/mydb/cassandra/bin/../lib/commons-lang3-3.1.jar:/mydb/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/mydb/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/mydb/cassandra/bin/../lib/disruptor-3.0.1.jar:/mydb/cassandra/bin/../lib/guava-15.0.jar:/mydb/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/mydb/cassandra/bin/../lib/jackson-core-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jackson-mapper-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar:/mydb/cassandra/bin/../lib/jbcrypt-0.3m.jar:/mydb/cassandra/bin/../lib/jline-1.0.jar:/mydb/cassandra/bin/../lib/json-simple-1.1.jar:/mydb/cassandra/bin/../lib/libthrift-0.9.1.jar:/mydb/cassandra/bin/../lib/log4j-1.2.16.jar:/mydb/cassandra/bin/../lib/lz4-1.2.0.jar:/mydb/cassandra/bin/../lib/metrics-core-2.2.0.jar:/mydb/cassandra/bin/../lib/netty-3.6.6.Final.jar:/mydb/cassandra/bin/../lib/reporter-config-2.1.0.jar:/mydb/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/mydb/cassandra/bin/../lib/slf4j-api-1.7.2.jar:/mydb/cassandra/bin/../lib/slf4j-log4j12-1.7.2.jar:/mydb/cassandra/bin/../lib/snakeyaml-1.11.jar:/mydb/cassandra/bin/../lib/snappy-java-1.0.5.jar:/mydb/cassandra/bin/../lib/snaptree-0.1.jar:/mydb/cassandra/bin/../lib/super-csv-2.1.0.jar:/mydb/cassandra/bin/../lib/thrift-server-0.3.3.jar
>>> org.apache.cassandra.service.CassandraDaemon
>>>
>>
>>
>>
>> On Mon, May 5, 2014 at 1:02 PM, Philip Persad <ph...@gmail.com>wrote:
>>
>>> Have you tried running "ulimit -a" as the Cassandra user instead of as
>>> root? It is possible that your configured a high file limit for root but
>>> not for the user running the Cassandra process.
>>>
>>>
>>> On Sun, May 4, 2014 at 6:07 PM, Yatong Zhang <bl...@gmail.com>wrote:
>>>
>>>> [root@storage5 ~]# lsof -n | grep java | wc -l
>>>>> 5103
>>>>> [root@storage5 ~]# lsof | wc -l
>>>>> 6567
>>>>
>>>>
>>>> It's mentioned in previous mail:)
>>>>
>>>>
>>>> On Mon, May 5, 2014 at 9:03 AM, nash <na...@gmail.com> wrote:
>>>>
>>>>> The lsof command or /proc can tell you how many open files it has. How
>>>>> many is it?
>>>>>
>>>>> --nash
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Cassandra 2.0.7 always failes due to 'too may open files' error
Posted by Yatong Zhang <bl...@gmail.com>.
I am running 'repair' when the error occurred. And just a few days before I
changed the compaction strategy to 'leveled'. don know if this helps
On Mon, May 5, 2014 at 1:10 PM, Yatong Zhang <bl...@gmail.com> wrote:
> Cassandra is running as root
>
> [root@storage5 ~]# ps aux | grep java
>> root 1893 42.0 24.0 7630664 3904000 ? Sl 10:43 60:01 java -ea
>> -javaagent:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar
>> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities
>> -XX:ThreadPriorityPolicy=42 -Xms3959M -Xmx3959M -Xmn400M
>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003
>> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
>> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:+UseTLAB -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true
>> -Dcom.sun.management.jmxremote.port=7199
>> -Dcom.sun.management.jmxremote.ssl=false
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Dlog4j.configuration=log4j-server.properties
>> -Dlog4j.defaultInitOverride=true -Dcassandra-pidfile=/var/run/cassandra.pid
>> -cp
>> /mydb/cassandra/bin/../conf:/mydb/cassandra/bin/../build/classes/main:/mydb/cassandra/bin/../build/classes/thrift:/mydb/cassandra/bin/../lib/antlr-3.2.jar:/mydb/cassandra/bin/../lib/apache-cassandra-2.0.7.jar:/mydb/cassandra/bin/../lib/apache-cassandra-clientutil-2.0.7.jar:/mydb/cassandra/bin/../lib/apache-cassandra-thrift-2.0.7.jar:/mydb/cassandra/bin/../lib/commons-cli-1.1.jar:/mydb/cassandra/bin/../lib/commons-codec-1.2.jar:/mydb/cassandra/bin/../lib/commons-lang3-3.1.jar:/mydb/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/mydb/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/mydb/cassandra/bin/../lib/disruptor-3.0.1.jar:/mydb/cassandra/bin/../lib/guava-15.0.jar:/mydb/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/mydb/cassandra/bin/../lib/jackson-core-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jackson-mapper-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar:/mydb/cassandra/bin/../lib/jbcrypt-0.3m.jar:/mydb/cassandra/bin/../lib/jline-1.0.jar:/mydb/cassandra/bin/../lib/json-simple-1.1.jar:/mydb/cassandra/bin/../lib/libthrift-0.9.1.jar:/mydb/cassandra/bin/../lib/log4j-1.2.16.jar:/mydb/cassandra/bin/../lib/lz4-1.2.0.jar:/mydb/cassandra/bin/../lib/metrics-core-2.2.0.jar:/mydb/cassandra/bin/../lib/netty-3.6.6.Final.jar:/mydb/cassandra/bin/../lib/reporter-config-2.1.0.jar:/mydb/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/mydb/cassandra/bin/../lib/slf4j-api-1.7.2.jar:/mydb/cassandra/bin/../lib/slf4j-log4j12-1.7.2.jar:/mydb/cassandra/bin/../lib/snakeyaml-1.11.jar:/mydb/cassandra/bin/../lib/snappy-java-1.0.5.jar:/mydb/cassandra/bin/../lib/snaptree-0.1.jar:/mydb/cassandra/bin/../lib/super-csv-2.1.0.jar:/mydb/cassandra/bin/../lib/thrift-server-0.3.3.jar
>> org.apache.cassandra.service.CassandraDaemon
>>
>
>
>
> On Mon, May 5, 2014 at 1:02 PM, Philip Persad <ph...@gmail.com>wrote:
>
>> Have you tried running "ulimit -a" as the Cassandra user instead of as
>> root? It is possible that your configured a high file limit for root but
>> not for the user running the Cassandra process.
>>
>>
>> On Sun, May 4, 2014 at 6:07 PM, Yatong Zhang <bl...@gmail.com> wrote:
>>
>>> [root@storage5 ~]# lsof -n | grep java | wc -l
>>>> 5103
>>>> [root@storage5 ~]# lsof | wc -l
>>>> 6567
>>>
>>>
>>> It's mentioned in previous mail:)
>>>
>>>
>>> On Mon, May 5, 2014 at 9:03 AM, nash <na...@gmail.com> wrote:
>>>
>>>> The lsof command or /proc can tell you how many open files it has. How
>>>> many is it?
>>>>
>>>> --nash
>>>>
>>>
>>>
>>
>
Re: Cassandra 2.0.7 always failes due to 'too may open files' error
Posted by Yatong Zhang <bl...@gmail.com>.
Cassandra is running as root
[root@storage5 ~]# ps aux | grep java
> root 1893 42.0 24.0 7630664 3904000 ? Sl 10:43 60:01 java -ea
> -javaagent:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar
> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42 -Xms3959M -Xmx3959M -Xmn400M
> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true
> -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dlog4j.configuration=log4j-server.properties
> -Dlog4j.defaultInitOverride=true -Dcassandra-pidfile=/var/run/cassandra.pid
> -cp
> /mydb/cassandra/bin/../conf:/mydb/cassandra/bin/../build/classes/main:/mydb/cassandra/bin/../build/classes/thrift:/mydb/cassandra/bin/../lib/antlr-3.2.jar:/mydb/cassandra/bin/../lib/apache-cassandra-2.0.7.jar:/mydb/cassandra/bin/../lib/apache-cassandra-clientutil-2.0.7.jar:/mydb/cassandra/bin/../lib/apache-cassandra-thrift-2.0.7.jar:/mydb/cassandra/bin/../lib/commons-cli-1.1.jar:/mydb/cassandra/bin/../lib/commons-codec-1.2.jar:/mydb/cassandra/bin/../lib/commons-lang3-3.1.jar:/mydb/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/mydb/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/mydb/cassandra/bin/../lib/disruptor-3.0.1.jar:/mydb/cassandra/bin/../lib/guava-15.0.jar:/mydb/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/mydb/cassandra/bin/../lib/jackson-core-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jackson-mapper-asl-1.9.2.jar:/mydb/cassandra/bin/../lib/jamm-0.2.5.jar:/mydb/cassandra/bin/../lib/jbcrypt-0.3m.jar:/mydb/cassandra/bin/../lib/jline-1.0.jar:/mydb/cassandra/bin/../lib/json-simple-1.1.jar:/mydb/cassandra/bin/../lib/libthrift-0.9.1.jar:/mydb/cassandra/bin/../lib/log4j-1.2.16.jar:/mydb/cassandra/bin/../lib/lz4-1.2.0.jar:/mydb/cassandra/bin/../lib/metrics-core-2.2.0.jar:/mydb/cassandra/bin/../lib/netty-3.6.6.Final.jar:/mydb/cassandra/bin/../lib/reporter-config-2.1.0.jar:/mydb/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/mydb/cassandra/bin/../lib/slf4j-api-1.7.2.jar:/mydb/cassandra/bin/../lib/slf4j-log4j12-1.7.2.jar:/mydb/cassandra/bin/../lib/snakeyaml-1.11.jar:/mydb/cassandra/bin/../lib/snappy-java-1.0.5.jar:/mydb/cassandra/bin/../lib/snaptree-0.1.jar:/mydb/cassandra/bin/../lib/super-csv-2.1.0.jar:/mydb/cassandra/bin/../lib/thrift-server-0.3.3.jar
> org.apache.cassandra.service.CassandraDaemon
>
On Mon, May 5, 2014 at 1:02 PM, Philip Persad <ph...@gmail.com>wrote:
> Have you tried running "ulimit -a" as the Cassandra user instead of as
> root? It is possible that your configured a high file limit for root but
> not for the user running the Cassandra process.
>
>
> On Sun, May 4, 2014 at 6:07 PM, Yatong Zhang <bl...@gmail.com> wrote:
>
>> [root@storage5 ~]# lsof -n | grep java | wc -l
>>> 5103
>>> [root@storage5 ~]# lsof | wc -l
>>> 6567
>>
>>
>> It's mentioned in previous mail:)
>>
>>
>> On Mon, May 5, 2014 at 9:03 AM, nash <na...@gmail.com> wrote:
>>
>>> The lsof command or /proc can tell you how many open files it has. How
>>> many is it?
>>>
>>> --nash
>>>
>>
>>
>
Re: Cassandra 2.0.7 always failes due to 'too may open files' error
Posted by Philip Persad <ph...@gmail.com>.
Have you tried running "ulimit -a" as the Cassandra user instead of as
root? It is possible that your configured a high file limit for root but
not for the user running the Cassandra process.
On Sun, May 4, 2014 at 6:07 PM, Yatong Zhang <bl...@gmail.com> wrote:
> [root@storage5 ~]# lsof -n | grep java | wc -l
>> 5103
>> [root@storage5 ~]# lsof | wc -l
>> 6567
>
>
> It's mentioned in previous mail:)
>
>
> On Mon, May 5, 2014 at 9:03 AM, nash <na...@gmail.com> wrote:
>
>> The lsof command or /proc can tell you how many open files it has. How
>> many is it?
>>
>> --nash
>>
>
>
Re: Cassandra 2.0.7 always failes due to 'too may open files' error
Posted by Yatong Zhang <bl...@gmail.com>.
>
> [root@storage5 ~]# lsof -n | grep java | wc -l
> 5103
> [root@storage5 ~]# lsof | wc -l
> 6567
It's mentioned in previous mail:)
On Mon, May 5, 2014 at 9:03 AM, nash <na...@gmail.com> wrote:
> The lsof command or /proc can tell you how many open files it has. How
> many is it?
>
> --nash
>
Re: Cassandra 2.0.7 always failes due to 'too may open files' error
Posted by nash <na...@gmail.com>.
The lsof command or /proc can tell you how many open files it has. How many
is it?
--nash