Open File Handles for Deleted sstables

[Anuj Wadehra <an...@yahoo.co.in>]

Hi,
We are facing an issue where Cassandra has open file handles for deleted sstable files. These open file handles keep on increasing with time and eventually lead to disk crisis. This is visible via lsof command. 
There are no Exceptions in logs.We are suspecting a race condition where compactions/repairs and reads are done on same sstable. I have gone through few JIRAs but somehow not able to coorelate the issue with those tickets. 
We are using 2.0.14. OS is Red Hat Linux.
Any suggestions?

ThanksAnuj

Re: Open File Handles for Deleted sstables

[Jeff Jirsa <je...@crowdstrike.com>]

There have been a history of leaks where repairs are multiple repairs were run on the same node at the same time ( e.g: https://issues.apache.org/jira/browse/CASSANDRA-11215 ) 

 

You’re running a very old version of Cassandra. If you’re able to upgrade to newest 2.1 or 2.2, it’s likely that at least SOME similar bugs are addressed.  

 

 

 

From: Anuj Wadehra <an...@yahoo.co.in>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Date: Wednesday, September 28, 2016 at 12:15 PM
To: User <us...@cassandra.apache.org>
Subject: Open File Handles for Deleted sstables

 

Hi, 

 

We are facing an issue where Cassandra has open file handles for deleted sstable files. These open file handles keep on increasing with time and eventually lead to disk crisis. This is visible via lsof command. 

 

There are no Exceptions in logs.We are suspecting a race condition where compactions/repairs and reads are done on same sstable. I have gone through few JIRAs but somehow not able to coorelate the issue with those tickets. 

 

We are using 2.0.14. OS is Red Hat Linux.

 

Any suggestions?

 

Thanks

Anuj

 

____________________________________________________________________
CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential and may be legally privileged. If you are not the intended recipient, do not disclose, copy, distribute, or use this email or any attachments. If you have received this in error please let the sender know and then delete the email and all attachments.

Re: Open File Handles for Deleted sstables

[Anuj Wadehra <an...@yahoo.co.in>]

Restarting may be a temporary workaround but cant be a permanent solution. After some days, the problem will come back again.
ThanksAnuj



Sent from Yahoo Mail on Android 
 
  On Thu, 29 Sep, 2016 at 12:54 AM, sai krishnam raju potturi<ps...@gmail.com> wrote:   restarting the cassandra service helped get rid of those files in our situation.
thanksSai
On Wed, Sep 28, 2016 at 3:15 PM, Anuj Wadehra <an...@yahoo.co.in> wrote:

Hi,
We are facing an issue where Cassandra has open file handles for deleted sstable files. These open file handles keep on increasing with time and eventually lead to disk crisis. This is visible via lsof command. 
There are no Exceptions in logs.We are suspecting a race condition where compactions/repairs and reads are done on same sstable. I have gone through few JIRAs but somehow not able to coorelate the issue with those tickets. 
We are using 2.0.14. OS is Red Hat Linux.
Any suggestions?

ThanksAnuj




  

Re: Open File Handles for Deleted sstables

[sai krishnam raju potturi <ps...@gmail.com>]

restarting the cassandra service helped get rid of those files in our
situation.

thanks
Sai

On Wed, Sep 28, 2016 at 3:15 PM, Anuj Wadehra <an...@yahoo.co.in>
wrote:

> Hi,
>
> We are facing an issue where Cassandra has open file handles for deleted
> sstable files. These open file handles keep on increasing with time and
> eventually lead to disk crisis. This is visible via lsof command.
>
> There are no Exceptions in logs.We are suspecting a race condition where
> compactions/repairs and reads are done on same sstable. I have gone through
> few JIRAs but somehow not able to coorelate the issue with those tickets.
>
> We are using 2.0.14. OS is Red Hat Linux.
>
> Any suggestions?
>
> Thanks
> Anuj
>
>
>