You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yang Yang (Commented) (JIRA)" <ji...@apache.org> on 2011/10/03 01:42:33 UTC
[jira] [Commented] (CASSANDRA-3269) accumulated 100GB of commit
logs
[ https://issues.apache.org/jira/browse/CASSANDRA-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119106#comment-13119106 ]
Yang Yang commented on CASSANDRA-3269:
--------------------------------------
the following shows that createNewSegment() sees that for the oldest segment file xxxxx63739.log , it's forcing out a flush of LocationInfo CF. it does this multiple times (at least 1000 times ... ), but the actualy CFS.forceFlush() always says that the CF is clean. so no flushing happens.
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 12:29:22,217 CommitLog.java (line 572) create new segment
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 12:29:22,217 CommitLog.java (line 523) commit log total size now:23488116331
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 12:29:22,217 CommitLog.java (line 578) forcing a flush on segments : CommitLog-1317524063739.log out of 176
WARN [COMMIT-LOG-WRITER] 2011-10-02 12:29:22,217 ColumnFamilyStore.java (line 718) forceFlush requested but everything is clean
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 12:29:22,218 CommitLog.java (line 584) forcing out CF:LocationInfo
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 12:30:15,357 CommitLog.java (line 572) create new segment
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 12:30:15,357 CommitLog.java (line 523) commit log total size now:23622334131
but when flushing (triggered by other cf) happens, the discardSegment() says that the oldest one is still dirty on LocationInfo.
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 05:11:14,182 CommitLog.java (line 502) Not safe to delete commit log CommitLogSegment(/mnt/cass/lib/cassandra/commitlog/CommitLog-1317524063739.log); dirty is opsMetrics (1001), LocationInfo (0), Budget (1000), ; hasNext: true
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 06:21:26,481 CommitLog.java (line 502) Not safe to delete commit log CommitLogSegment(/mnt/cass/lib/cassandra/commitlog/CommitLog-1317524063739.log); dirty is LocationInfo (0), Budget (1000), ; hasNext: true
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 06:21:26,868 CommitLog.java (line 502) Not safe to delete commit log CommitLogSegment(/mnt/cass/lib/cassandra/commitlog/CommitLog-1317524063739.log); dirty is LocationInfo (0), ; hasNext: true
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 06:32:15,341 CommitLog.java (line 502) Not safe to delete commit log CommitLogSegment(/mnt/cass/lib/cassandra/commitlog/CommitLog-1317524063739.log); dirty is LocationInfo (0), ; hasNext: true
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 06:32:20,905 CommitLog.java (line 502) Not safe to delete commit log CommitLogSegment(/mnt/cass/lib/cassandra/commitlog/CommitLog-1317524063739.log); dirty is LocationInfo (0), ; hasNext: true
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 06:32:39,725 CommitLog.java (line 502) Not safe to delete commit log CommitLogSegment(/mnt/cass/lib/cassandra/commitlog/CommitLog-1317524063739.log); dirty is LocationInfo (0), ; hasNext: true
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 06:32:54,023 CommitLog.java (line 502) Not safe to delete commit log CommitLogSegment(/mnt/cass/lib/cassandra/commitlog/CommitLog-1317524063739.log); dirty is LocationInfo (0), ; hasNext: true
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 06:33:01,934 CommitLog.java (line 502) Not safe to delete commit log CommitLogSegment(/mnt/cass/lib/cassandra/commitlog/CommitLog-1317524063739.log); dirty is LocationInfo (0), ; hasNext: true
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 06:33:11,167 CommitLog.java (line 502) Not safe to delete commit log CommitLogSegment(/mnt/cass/lib/cassandra/commitlog/CommitLog-1317524063739.log); dirty is LocationInfo (0), ; hasNext: true
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 08:16:28,250 CommitLog.java (line 502) Not safe to delete commit log CommitLogSegment(/mnt/cass/lib/cassandra/commitlog/CommitLog-1317524063739.log); dirty is LocationInfo (0), ; hasNext: true
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 08:16:36,208 CommitLog.java (line 502) Not safe to delete commit log CommitLogSegment(/mnt/cass/lib/cassandra/commitlog/CommitLog-1317524063739.log); dirty is LocationInfo (0), ; hasNext: true
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 08:16:56,037 CommitLog.java (line 502) Not safe to delete commit log CommitLogSegment(/mnt/cass/lib/cassandra/commitlog/CommitLog-1317524063739.log); dirty is LocationInfo (0), ; hasNext: true
DEBUG [COMMIT-LOG-WRITER] 2011-10-02 08:17:18,003 CommitLog.java (line 502) Not safe to delete commit log CommitLogSegment(/mnt/cass/lib/cassandra/commitlog/CommitLog-1317524063739.log); dirty is LocationInfo (0), ; hasNext: true
since the isClean() of CFS is determined by columnFamilies.size() and isClean() of segment is determined by lastCFWrite(), it seems that these 2 data structures somehow got out of sync.
also an easy suspect is that the lastCFWrite is not concurrent, but I can't show an exact scenario of how this would lead to the symptom I see
> accumulated 100GB of commit logs
> --------------------------------
>
> Key: CASSANDRA-3269
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3269
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Yang Yang
>
> I ran my cluster for about 2 days. the cluster has 2 nodes. I restarted one box several times, and the other one was always running. the one always running ended up accumulating 100GB of commit logs.
> this is 1.0.0 code from about Sept 15 in github. I kept the original setting for
> #commitlog_total_space_in_mb: 4096
> i.e. commented out
> here is some sample of the output:
> -rw-r--r-- 1 yyang yyang 134217857 2011-09-28 03:51 CommitLog-1317181834810.log
> -rw-r--r-- 1 yyang yyang 134217869 2011-09-28 03:50 CommitLog-1317181764105.log
> -rw-r--r-- 1 yyang yyang 134217783 2011-09-28 03:49 CommitLog-1317181694633.log
> -rw-r--r-- 1 yyang yyang 134217750 2011-09-28 02:39 CommitLog-1317176955102.log
> yyang@ip-10-71-21-46:/mnt/cass/log/cassandra$ ls -lt /mnt/cass/lib//cassandra/commitlog/|wc -l
> 727
> yyang@ip-10-71-21-46:/mnt/cass/log/cassandra$ du -s /mnt/cass/lib/cassandra/commitlog/
> 95095316 /mnt/cass/lib/cassandra/commitlog/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira