You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Illarion Kovalchuk (JIRA)" <ji...@apache.org> on 2013/03/06 09:42:13 UTC
[jira] [Commented] (CASSANDRA-5308) Cassandra with leveled
compaction quickly runs out of free space with tons of tmp files
[ https://issues.apache.org/jira/browse/CASSANDRA-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594495#comment-13594495 ]
Illarion Kovalchuk commented on CASSANDRA-5308:
-----------------------------------------------
After restarting the problematic node, I found that it starts to grow the tmp files again, and nodetool tpstats shows, that it has some problems with FlushWriter: less completed, and more blocked all the time...
./onall.sh "nodetool tpstats | grep FlushWriter"
Pool Name Active Pending Completed Blocked All time blocked
122dev-CAFS1 ~: FlushWriter 0 0 15891 0 16
122dev-CAFS2 ~: FlushWriter 0 0 14211 0 18
122dev-CAFS3 ~: FlushWriter 0 0 371 0 38
122dev-CAFS4 ~: FlushWriter 0 0 12690 0 11
> Cassandra with leveled compaction quickly runs out of free space with tons of tmp files
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-5308
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5308
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.10
> Environment: Cluster with 4 nodes running cassandra 1.1.10
> Reporter: Illarion Kovalchuk
> Fix For: 1.1.11
>
> Attachments: cassandra-1.1.10-tmpfiles-symptoms.txt
>
>
> We were performing a massive upload of data to our cluster and found, that one of the nodes is ruuning out of free space
> nodetool ring didn't show any extra usage, but df -h / on the problamatic node shown us 95% of used space, while other nodes were all around 50%
> Eventually, we discovered that there's an abnormous peak of tmp files in one of our CF's: 753'906'319'257 bytes of data in 26811 files.
> Rebooting the node caused all that files to get deleted and node went back to normal.
> One of the symptoms: when the node started to misbehave and collect the temp files, it started to flush the memtables due to lot of heap used.
> See attached info
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira