[jira] [Comment Edited] (CASSANDRA-7042) Disk space growth until restart

["Zach Aller (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745796#comment-14745796 ] 

Zach Aller edited comment on CASSANDRA-7042 at 9/15/15 5:40 PM:
----------------------------------------------------------------

We are not experiencing the issue anymore it was fixed on some version in the 2.0.x line and we are now on 2.1.9 so it fixed in both versions. If my memory serves me right i could be way wrong i think it was the above patches that fixed it also i think i remember building cassandra with those and it worked.


was (Author: zaller):
We are not experiencing the issue anymore it was fixed on some version in the 2.0.x line and we are now on 2.1.9 so it fixed in both versions.

> Disk space growth until restart
> -------------------------------
>
>                 Key: CASSANDRA-7042
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>            Reporter: Zach Aller
>         Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png, after.log, before.log, system.log.9.gz, tabledump_after_restart.txt, tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only thing that seems to fix it is a restart of cassandra this happens about every 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we restart cassandra it usually all clears itself up and disks return to normal for a while then something triggers its and starts climbing again. Sometimes when we restart compactions pending skyrocket and if we restart a second time the compactions pending drop off back to a normal level. One other thing to note is the space is not free'd until cassandra starts back up and not when shutdown.
> I will get a clean log of before and after restarting next time it happens and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db (No such file or directory)
>         at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
>         at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
>         at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
>         at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
>         at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
>         at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
>         at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException: /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db (No such file or directory)
>         at java.io.RandomAccessFile.open(Native Method)
>         at java.io.RandomAccessFile.<init>(Unknown Source)
>         at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
>         at org.apache.cassandra.io.util.ThrottledReader.<init>(ThrottledReader.java:35)
>         at org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
>         ... 17 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)