Reading SSTables Potential File Descriptor Leak 1.2.18

[Tim Heckman <ti...@pagerduty.com>]

Hello,

I ran in to a problem today where Cassandra 1.2.18 exhausted its number of
permitted open file descriptors (65,535). This node has 256 tokens (vnodes)
and runs in a test environment with relatively little traffic/data.

As best I could tell, the majority of the file descriptors open were for a
single SSTable '.db' file. Looking in the error logs I found quite a few
exceptions that looked to have been identical:

ERROR [ReadStage:3817] 2014-09-19 07:00:11,056 CassandraDaemon.java (line
191) Exception in thread Thread[ReadStage:3817,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to
seek to position 29049 in /mnt/var/lib/cassandra/data/path/to/file.db (1855
bytes) in read-only mode

Upon further investigation, it turns out this file became 'read-only' after
the Cassandra node was gracefully restarted last week. I'd imagine this is
a discussion for another email thread.

I fixed the issue by running:

nodetool scrub <Keyspace>
nodetool repair <Keyspace>

Attached to this email is one of the log entries/stacktrace for this
exception.

Before opening a JIRA ticket I thought I'd reach out to the list to see if
anyone has seen any similar behavior as well as do a bit of source-diving
to try and verify that the descriptor is actually leaking.

Cheers!
-Tim

Re: Reading SSTables Potential File Descriptor Leak 1.2.18

[Robert Coli <rc...@eventbrite.com>]

On Tue, Sep 23, 2014 at 5:47 PM, Tim Heckman <ti...@pagerduty.com> wrote:

> As best I could tell, the majority of the file descriptors open were for a
> single SSTable '.db' file. Looking in the error logs I found quite a few
> exceptions that looked to have been identical:
>
...

> Before opening a JIRA ticket I thought I'd reach out to the list to see if
> anyone has seen any similar behavior as well as do a bit of source-diving
> to try and verify that the descriptor is actually leaking.
>

I would (search for, and failing to find one..) open a JIRA, and let the
list know its URL.

=Rob

RE: Reading SSTables Potential File Descriptor Leak 1.2.18

[Job Thomas <jo...@suntecgroup.com>]

Hi, It look like the offset in keycache is wrong !!.
 
 refresh the keycache may solve the issue.
 
Thanks & Regards
Job M Thomas 
Platform & Technology

________________________________

From: Tim Heckman [mailto:tim@pagerduty.com]
Sent: Wed 9/24/2014 6:17 AM
To: user@cassandra.apache.org
Subject: Reading SSTables Potential File Descriptor Leak 1.2.18


Hello,

I ran in to a problem today where Cassandra 1.2.18 exhausted its number of permitted open file descriptors (65,535). This node has 256 tokens (vnodes) and runs in a test environment with relatively little traffic/data.

As best I could tell, the majority of the file descriptors open were for a single SSTable '.db' file. Looking in the error logs I found quite a few exceptions that looked to have been identical: 

ERROR [ReadStage:3817] 2014-09-19 07:00:11,056 CassandraDaemon.java (line 191) Exception in thread Thread[ReadStage:3817,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to seek to position 29049 in /mnt/var/lib/cassandra/data/path/to/file.db (1855 bytes) in read-only mode

Upon further investigation, it turns out this file became 'read-only' after the Cassandra node was gracefully restarted last week. I'd imagine this is a discussion for another email thread.

I fixed the issue by running:

nodetool scrub <Keyspace>
nodetool repair <Keyspace>

Attached to this email is one of the log entries/stacktrace for this exception.

Before opening a JIRA ticket I thought I'd reach out to the list to see if anyone has seen any similar behavior as well as do a bit of source-diving to try and verify that the descriptor is actually leaking.

Cheers!
-Tim