[jira] [Created] (CASSANDRA-11985) 2.0.x leaks file handles (Again)

["Amit Singh Chowdhery (JIRA)" <ji...@apache.org>]

Amit Singh Chowdhery created CASSANDRA-11985:
------------------------------------------------

             Summary: 2.0.x leaks file handles (Again)
                 Key: CASSANDRA-11985
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11985
             Project: Cassandra
          Issue Type: Bug
          Components: Compaction
         Environment: Unix kernel-2.6.32-431.56.1.el6.x86_64, Cassandra 2.0.14
            Reporter: Amit Singh Chowdhery
            Priority: Critical


We are running Cassandra 2.0.14 in production environment and disk usage is very high. On investigating it further we found that there are around 4-5 files(~ 150 GB) in stuck mode.

Command Fired : lsof /var/lib/cassandra | grep -i deleted 

Output : 

java    12158 cassandra  308r   REG   8,16  34396638044 12727268 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-16481-Data.db (deleted)
java    12158 cassandra  327r   REG   8,16 101982374806 12715102 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-126861-Data.db (deleted)
java    12158 cassandra  339r   REG   8,16  12966304784 12714010 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-213548-Data.db (deleted)
java    12158 cassandra  379r   REG   8,16  15323318036 12714957 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-182936-Data.db (deleted)

we are not able to see these files in any directory. This is somewhat similar to CASSANDRA-6275 which is fixed but still issue is there on higher version. Also in logs no error related to compaction is reported.

so could any one of you please provide any suggestion how to counter this. Restarting Cassandra is one solution but this issue keeps on occurring so we cannot restart production machine is not recommended so frequently.

Also we know that this version is not supported but there is high probability that it can occur in higher version too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)