RE: C* files getting stuck

[Amit Singh F <am...@ericsson.com>]

Hi All,

Please check if anybody has faced below issue and if yes what best can be done to avoid this.?
Thanks in advance.

Regards
Amit Singh

From: Amit Singh F [mailto:amit.f.singh@ericsson.com]
Sent: Wednesday, June 29, 2016 3:52 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: C* files getting stuck


Hi All

We are running Cassandra 2.0.14 and disk usage is very high. On investigating it further we found that there are around 4-5 files(~ 150 GB) in stuck mode.

Command Fired : lsof /var/lib/cassandra | grep -i deleted

Output :

java 12158 cassandra 308r REG 8,16 34396638044 12727268 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-16481-Data.db (deleted)
java 12158 cassandra 327r REG 8,16 101982374806 12715102 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-126861-Data.db (deleted)
java 12158 cassandra 339r REG 8,16 12966304784 12714010 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-213548-Data.db (deleted)
java 12158 cassandra 379r REG 8,16 15323318036 12714957 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-182936-Data.db (deleted)

we are not able to see these files in any directory. This is somewhat similar to  https://issues.apache.org/jira/browse/CASSANDRA-6275 which is fixed but still issue is there on higher version. Also in logs no error related to compaction is reported.

so could any one of you please provide any suggestion how to counter this. Restarting Cassandra is one solution but this issue keeps on occurring so we cannot restart production machine is not recommended so frequently.

Also we know that this version is not supported but there is high probability that it can occur in higher version too.
Regards
Amit Singh

Re: C* files getting stuck

[Paul Fife <pa...@gmail.com>]

Hello Amit -

I can confirm that we also experienced this issue in 2.0.x and were not
able to find a solution other than a restart. Since upgrading to 2.2.x the
problem did disappear.

Thanks,
Paul Fife

On Fri, Sep 30, 2016 at 6:48 AM, Amit Singh F <am...@ericsson.com>
wrote:

> Hi All,
>
>
>
> Please check if anybody has faced below issue and if yes what best can be
> done to avoid this.?
>
> Thanks in advance.
>
>
>
> Regards
>
> Amit Singh
>
>
>
> *From:* Amit Singh F [mailto:amit.f.singh@ericsson.com
> <am...@ericsson.com>]
> *Sent:* Wednesday, June 29, 2016 3:52 PM
> *To:* user@cassandra.apache.org
> *Subject:* C* files getting stuck
>
>
>
> Hi All
>
> We are running Cassandra 2.0.14 and disk usage is very high. On
> investigating it further we found that there are around 4-5 files(~ 150 GB)
> in stuck mode.
>
> Command Fired : lsof /var/lib/cassandra | grep -i deleted
>
> Output :
>
> java 12158 cassandra 308r REG 8,16 34396638044 12727268
> /var/lib/cassandra/data/mykeyspace/mycolumnfamily/
> mykeyspace-mycolumnfamily-jb-16481-Data.db (deleted)
> java 12158 cassandra 327r REG 8,16 101982374806 12715102
> /var/lib/cassandra/data/mykeyspace/mycolumnfamily/
> mykeyspace-mycolumnfamily-jb-126861-Data.db (deleted)
> java 12158 cassandra 339r REG 8,16 12966304784 12714010
> /var/lib/cassandra/data/mykeyspace/mycolumnfamily/
> mykeyspace-mycolumnfamily-jb-213548-Data.db (deleted)
> java 12158 cassandra 379r REG 8,16 15323318036 12714957
> /var/lib/cassandra/data/mykeyspace/mycolumnfamily/
> mykeyspace-mycolumnfamily-jb-182936-Data.db (deleted)
>
> we are not able to see these files in any directory. This is somewhat
> similar to  https://issues.apache.org/jira/browse/CASSANDRA-6275 which is
> fixed but still issue is there on higher version. Also in logs no error
> related to compaction is reported.
>
> so could any one of you please provide any suggestion how to counter this.
> Restarting Cassandra is one solution but this issue keeps on occurring so
> we cannot restart production machine is not recommended so frequently.
>
> Also we know that this version is not supported but there is high
> probability that it can occur in higher version too.
>
> Regards
>
> Amit Singh
>